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THE WORK STATEMENT 


THE WORK STATEMENT 

This proposal consists of four oajor components: 

M The Artificial Intelligence Laboratory proposes to work on 
issues central to the largo data base problem, the 
intelligent terminal problem, and. the natural language 
engineering problen with a broad, integrated research 
program, There are several imedlate tasks: 

To pfoduce a natural language interface for accessing 
intelligent terminal and large data base tools. 

To produce a natural language generating program to act with 
the understanding program. 

To develop JlinsXy's frame system, into a fora suitable for 
dealing with large: bodies of knowledge. 

To create resaurs-a allocation proarits suitable for a 
variety of scheduling tasks. 

To create models of people and organizations such that 
programs can understand their needs, 

i The Artificial Intelligence Laboratory proposes to *ork on 
applications of computer vision with the emphasis shifting 
from tool building toward solving real problems. Several 


new tasks are to be undertaken; 


To create an. inspection program for dealing with 
sophisticated Jet engine castings. 

To create representations for terrain such that vehicles can 
discover their own orientation using it. 

t 

To develop tracking jrograns that use knowledge about the 
objects tracked beyond simple velocity and acceleration 
prediction. 

■ The Artificial Intelligence Laboratory proposes to continue 
a basic studies program aimed at dealing with deep, central 
issues. The issues ore to he exposed by work in computer 
vision, common sense reasoning, and expert problem solving. 
The ubiquitous representation problem is involved lb ill 
throe areas. The basic goals include the following;. 

To create solid symbolic representations from loag&S, 

Tc create programs which can do Comaon sense reasoning about 
the simple physical world. 

To create expert problem solvers that expose the general 
issues pervasive in the creation of all engineered systems. 

■ The Artificial intelligen.ee Laboratory proposes to complete 
development of low cast hardware intended to servo single 
users with an outstanding LISP computing system dramatically 
superior to any available today both in terms of power and 


cost, 


There is tu be 
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INTRODUCTION 
TO THE 
PROPOSAL 


THE PHOPQBAL 


This proposal specifies both our long terra direction end our specific goals 
as planned! Tor the riant two years, We relate these goals to past progress r 
tuned late practical needs,, and long tern scientific objectives. Since many 
of the scientific objectives cited are deep and demanding, readers should 
understand that considerably core than two years will he required for full 
solution. 

THE FOUNDATIONS 

From the beginning our purpose has been to make machines pore intelligent, 
partly to make them more useful and partly to learn about the nature of 
intelligence. Certainly the need to make machines smarter is clear. It is 
also clear- that the ideas of Artificial intelligence are the principle 
scientific path toward making! machines smarter, tfom soon Util it happen and 

Aon JTRTC.fr ml i l it CDS t? 

What aakos prediction difficult, of course, is the non-linear 
relationship between the level of technical development and the payoff. For 
any combination of Artificial Intelligence paired with a need, there is some 
threshold on costs and capabilities such that on one side, little happens, 
and on the other, revolution. The pocket calculator phenomenon is one 
example of this that it known to everyone. The pocket computer is around 
the corner. We believe that there are many areas in which Artificial 
Intelligence will produce its own revolutions reselling that of the pccket 
calculator In their swiftness. Two particularly promising areas are these: 

i The combination of large data base handling, intelligent 
interfacing of software tools and people, and natural 
language interface engineering. 


This revolution will be based on a qualitative advance in which programs. 
uftdtrt tand , frg & significant degree, the information they are handlings the 
current technology merely an plies manipulative rituals that do hot use much 
of the fcnowldege embedded in the data bases baing manipulated. 

£ The application of the ideas of artificial intelligence to 
computer vision and manipulit ion for productivity 
technology! vehicle guidance, and image understanding. 

Again, the breakthrough will depend on the use of representations of the 
noenings, uses, and special features of the objects to be sensed and 
Danipulated. 

Those are only what we view as Immediate ureas of maximum opportunity. 
Many others are suggested by the following sample; 


SOME AREAS OF APPLICATION 


Manufacturing 


Assembling Electronic Systems 
Assembling Snail Mechanical Devices 
Coal Mining Cn Dangerous Ninos 
Undersea Recovery Of Manganese Modules 
Selective Harvesting With Mixed Crops 
Past Control 


Mining 


Farming 


Pruning 


Repair And Maintenance 


Vehicle Debugging 


Floor Cara 


Purposeful JfDnitartng 


Automatic Pr&grammipg 


Management Assistance* 


Logistics 


Medicine ' 


Education 


Fire Fishting 


Theft Prevention 


intelligent Autopilots 
Shipboard Functions 
Management Reports 
Inventory Control 
Production Scheduling 
Quality Coot rol 
Very Large Data Bases 
Intelligent PERI Networks 
Schfi dull tig Recple end Groups 
Question Referral 
Ke^'S Summarising 
Docunent Draft Polishing 
Routing 

Intelligent Substitution 
Equipment Sharing 
Diagnosis 

Intensive care Monitoring 
Treatment Management 
Radiation Therapy Setup 
Intelligent CA1 Systems 
Information Retrieval 
Fearless Fireman 
Apparatus Allocation 
Trackers 


How can such objectives be roichedT Certainly for smart machines to make an 
impact in aruas like th&se mentioned, the snort nachines must ox'hiblt a. 
variety of tale-nts associated with intelligence- Computer vision, English 
understanding, learning and debugging, and export problem solving all must 
ba understood- It is natural then that considerable effort has gone into 


understanding thBse things, Happily we flet back to them both hy way of our 
desire to lay the foundation for applications and hy way of our hope to 
understand intelligence for its own sake, 

GENERAL OBJECTIVE: TO MCZ MACHINES SflARTER 


TO HAKE, MACHINES MORE USEfUL TO UNDERSTAND' INTELLIGENCE 


Areas Of Application 

Intelligent Terminals 
Vftry Large Tata Bases 
Remote Guidance 
Photo Interpretation 
Logistics 

Purposeful Monitoring 
Repair And Maintenance 
Cleaning 

Manufacturing and Mining 

Medicine 

Education 


Systems To Be (Jndarstood 

Image And Scene Analysis 
Speech And Text Understanding 
Learning and Debugging 
Expert Problem Solving 
Cciunon Sense Reasoning 
Goal. Understanding 


As the diagram suggests, it has been sensible during the gestation years to 
b« organized around theses like computer vision, English tmdestending. 


laariiing and debugging, ar-.d uxpert problem solving. It U through these 
particular thrusts that Wfc have learn ad A great deal about what are new 
regarded as central theoretical issues; 


SOME CENTRAL THtOR.E.T 1CAL ISSUES 


Knowledge Representalion 
Search And Centre 1 
Programming Methodology 
Grouping 

THE FUTURE 


Constraint Exploitation 
Classification And Deduction 
Goal Directed Versus Button Up 
ferial Versus Parallel 


rfte Structure Of Tttt EdDora10rtf It E^ofpfitg Wftfc The Science 

The Artificial Intelligence Laboratory is widely recognised as one of the 
world's principal sources of ideas, experimental prototypes, and feasibility 
dempnitrations on the frontier of computer science* It would not be 
appropriate for the laboratory to concentrate on producing operational 
ays tens in areas like that of very large data bases, but it does seem 
appropriate to begin thinking in terms of what AI can laad to in the near 
term and to begin producing the basic research results lying lust in .front 
Of prototype production. To this end, we nan see ourselves organized into a 
new structure that draws not only f™ the desire to understand various 
dimensions of intelligence but also from a morn direct interest in 
applications challenges on one side and an interest in the emerging central 
issues on the other. Manifestly the boundaries are to be soft, not hard, 
and ns has been cannon in the past, much flaw among the particular subgroups 
is expected. 


djplteotiofls Oriented flirsic studies 


Bccaus-s natural language research has mde much progress along with parallel 
advances in our theories of knowledge, a considerable portion of our 
resources should be assigned to a unified program on Very large data bases, 
intelligent terminals, and natural Language interface engineering. Part of 

our efforts for the reminder of 1375 will be directed at bringing this area 
dp to speed. 

AppLieotiOflS Oriented i/lsiorr 4 jio! fle a i pu J at i o it 

We have spent considerable time and effort in developing a laboratory 
facility that is at once powerful, easy to copy, and relatively inexpensive. 
It has been developed using electronic circuit board assembly as the test 
problem. It is now time to redirect the use of this facility to other 
pressing prnhlems by selecting the best of many alternatives for allocation 
of this important resource. Fully automatic assembly of small mechanical 
devices seeps appropriate, part of the image understanding problem also 
seems to be promising, 

/■SSJieS Orients fleslc Studies 

The study of basic issues with domains selected to facilitate understanding 
remains crucial to our progran. Within this collection of topics m see 
continued work, on the re pro sen cation of knowledge, perhaps our most pressing 
problem area, as well as on expert problem solving, common sense physical 
reasoning, problem solving by analogy, learning, and constraint driven 
problem solving of the sort exposed by studies in computer vision. This 
last mentioned work on computer vision is believed to be critical for long 
range progress on Visual interpretation problems that require nearly all of 


4 large array of image points to be processed. 


System dnd &lf-sier\& Con-cep£i 

Progress In Artificial Intelligence is facilitated by adequate equipment. 
Inclined,, many of its graat accom^lishne nts have Involved programs that are 
tgp large and to etna pi ax to he produced without the frontier time sharing 
concepts wo have developed. Jlfow time sharing is well established, but 
newer, even pare revolutionary opportunities have emerged, It therefore 
seems wise to continue our systems concepts leadership, both bEcavse of our 
own needs in Artificial Intelligence and because of the profound lop-act we 
believe these conputer ideas wilt soon have on tbs computer censuBing 
community, 

RECENT PROGRESS 

This introduction closes With a summary Of recent progress organized around 
the topics which correspond to our past feci. These artE 

Representing Knowledge and Expert Problem So lying 

Understanding Natural Language 

Understanding Vision 

Productivity Technology 

learning and Debugging 

Systems and Architecture 

and New Programs 


ftep re3©Jrttff V KmhdIe dpe Aetf Expert Problem So [virtu 


Sussman and his StudantS have nude progress in thn direction Of 
understanding debugging and the aspert problem solving process in the 
particular domain of electronic circuitry. Sussman and Stallman* for 
example, completed a program anil a paper in Narct or 19?5 a "Heuristic 
Techniques in Computer Aided Circuit Analysis.” The program understands 
circuits well enough to determine proper operating voltages and currants by 
common-sense reasoning, rather than by solving network equations. It can, 
now handle the rather complicated circuits found in IC Chips. Specifically, 
the circuitry of a UA741 op M.p hiS been successfully Analyzed. 'Experts In 
th e electronics field have shown considerable Interest in this flew, 
knowledge-baked, common sense approach. 5USS[B4H believes that tbo 
principles will transfer smoothly to ether doesins and become a general 
theory of debugging applicable to all engineered systems. 

Fahlman and Qrosssan have done extensive work on ft new representational 
scheme which w# ballava may have considerable consequences with respect to 
future machine architectures, Fahlsan and Crossman argue for a network of 
nodes and relationships from which knowledge is retrieved by parallel 
searches Along A-JtJND-Of and other links. Their thoughts have been 
geminating over the past year, bet the work was too preliminary to allow 
commitment to- writlhg until recently, kow both Fahlman end Grossman have 
completed papers which outline theories, propose experiments, and define, in 
the case of Grossman, specific application possibilities in tbs very large 
data base area. New hardware progress in the integrated circuit industry 
provides Some hope that machines oriented toward such searches may 'become 
practical, given that simulation on serial r.achines proves the Ideas sound. 

Sandewall has concentrated on the data base problem AS well, but With 
with a view toward dealing with sone liwediate problems of data base 
organization And. self description, He ha* worked out a block structured 


scheme capable of handling hoth procedure and data,, designed to smooth the 
interface between programs an ane side and a collection of data bases on the . 
other. It is expected that his evolving theory will particularly Influence 
our proposed work an very large data bases, Sandewall has finished a 
prototype system of programs that illustrate his ideas. 

Understanding Natural tonguefc 

Pratt and his students have concentrated on the problem Of dealing with 
efficiency issues in parsing natural language text. This effort takes us in 
the comfortable direction or preparing for the time when real application of 
natural language systems will make a theoretical understanding of tha 
efficiency issue of great practical importance. Pratt's paper of January, 
1975, presents an optimality proof demonstrating that blNGDL's parsing 
algorithm Is optimal among those of its class in the sense that all the 
phrases that It builds up cam be U.s#d. 

fleanwhtle Ha reus, has Completed the progrardfl lug of A now parsing system 
based on the Wait-and-see philosophy for handling ambiguity. Instead of 
picking one option at all decision points with the risk of being stuck with 
costly retreat, Pfarcus 1 system carries all options forward, making a 
decision only when the decision can be made with certainty. The design of 
this parser was finished by the and nf 1374; Implementation took place 
during 1975 and is Just now complete. The next task is to augnent the snail 
grammar If a reus has used in debugging his parser with a much more complete 
grammar, one large enough to be a good interface to various intolllgont 
terminal or large data base modules, 

Progress has also basil made on the relatively neglected linguistic 
problem of generating language. This involves, among other things, a need 
to know what the listener already knows. McDonald has completed an M.£. 
Thetis studying the problem. There is no program embodiment of the ideas it 


puts forth, but creating and exper loan tiny with such a program seems 
sensible and is planned. 

£i'ndis r^ t□ .nd l rjj Vijio.n 

Gene Preutier has developed a system for visual recognition, with 
implications for the general problem of control. The system is intended to 
provide the flexibility required Ed deal with realistic visual scenes; he 
is looking at htamers, net stylised models, but everyday, toolbox hammers, 
under natural conditions (e.g, lighting, background, orientation, 
occlusion) ► Ha is studying directly the interac11on of knowledge and 
control required in an integrated pass at the entire task, rather than 
looking at the attendant problems of image processing in detail. Particular 
results, as they are acquired, Combine With general visual knowledge to 
"suggest" and "advise" further processing, This process, called "active 
knowledge," extends end implements the principles of "heterarchy" and 
"domain directed processing" pioneered in this laboratory. 

Since description has lung beep regarded as the key to success with any 
difficult problem in artificial intelligence, Hdllerbaeh 1 s work on 
describing curved objects is another inportent step forward. Ho has 
developed an approach towards shape description based un prototype 
modification and generalised cylinders, a notion invented by Binfordi at 
Stanford, His program describe and identify pottery from vase outlines 
well enough to bo remarkably consistent with the descriptions and 
identifications given by archeologists. Hie resulting descriptions seem 
very natural and suitable far dealing with many sorts of objects besides 
pottery. The emphasis throughput has been to develop useful, qualitative 
descriptions which bring out the significant features and subordinate lesser 
ones, 

Harr and his students Continue te look at vision from another poiht of 



view, that of concentration on thoroughly understanding the problem of 
translating image arrays Into symbolic descriptions without the intervention 
of high level knowledge, Their thesis it that real inage understanding 
cannot be done without vary solid programs working at this levels This 
basic worlt seeds directly on the critical path leading toward real image 
understanding, be it of ordinary scenes or £ RTS type data- As of January h 
i9?S f the work had concentrated on discovering what features of one- 
dimensional intensity profiles could reliably provide inforoation about edge 
typos and such modifiers as sharpness of edge focus. Since then, the edge 
and edge modifier wort has been extended to two dimensions with satisfactory 
results- The nDst important work, however, has been on the problem of 
Urou.pln.gi the very lowest level symbolic descriptions of edges together into 
larger symbolic aggregates and the subsequent realisation that a new theory 
of texture will result from this wort. Tlarr has invented about four new 
grouping algorithms and prototype programs have been written for them. Some 
experiments with these prototype programs have been encouraging,, but for the 
experimentation needed, the programs will have to be rewritten for the sake 
of efficiency. Working with image arrays is tine consuming and requires 
more than the usual attention to program efficiency. 

Typical of other work directed by harr is a theory of visual light 
source detection devalopEd by L'llmdn* Several factors were examined t among- 
which were: absolute Intensity values, intensity compared with the average 
illumination in the scene, both local and global contrast f and lightness 
contrast. A surprising experimental result was that theso factors worn 
shown to be insufficient for explaining our ability to detect light sourcas 
in the visual field, oven under very simple conditions. While high enough 
contrast, for example, is a sufficient condition for creating the. perception 
of radiance, it was Shown that a light source can sometimes be detected in 
cases where the contrast is very low, A nethod Tor da tec ting light sources 
was than proposed and implemented, bftSftd on the Comparison bf two ratios: 


for 4 given pair of adjacent areas,, both intensities-ratlo and gradients- 
ratio are conputid. If neither of the areas is- a light source the ratios 
ihciuld be equal r if they aro not equal, one of the areas is a light source. 
The method car thereby detect the source and compute its intensity. 

A collection of Qhr previous Papers on vision, learning, and 
representation has just been published by ftctiraw-IUll, under the title. The 
Psychalogtj of Computer Lfisioj], Sales have been encouraging. This is part of 
an effort to disseminata our work and speed the technology transfer. 

P roductf V f Iff Technologg 

Electronic circuit problems ware selected as an initial focus for developing 
automatic nanipul atlon and inspection idEas because the parts are fairly 
well constrained to 4 few types and because the vision is somewhat more two 
dimensional than in most problem areas, 

Horn's program for visually aligning IC chips in preparation for lead 
bonding Was a Step forward since this Job is now donE entirely by human eye 
in spite of attempts made by industry t* automate the process. This work 
was largely completed in Septeobsr of 1&74. Since that time some polishing 
has taken place, and efforts have bean made to transfer the technology to 
practice. An article on this, "Orienting Silicon Integrated Circuit Chips 
for Lead Bonding.," has been accepted for pub licet ion in Computer GrppJitej 
□ ad fiturge processing. CCA, the leading maker of automatic bonding equipment, 
produces a iiichlna by which one person does the alignment for several 
bonding stations, but they have not been able to -eliminate the human 
alignment step. The need appears to be particularly acute in the 
manufacture of hybrid Circuits which Should not be manufactured by foreign 
suppliers, 

faartzar has worked cn inspection higher up at the levs! of solder blob 
checking. Unlike Horn's prograa, Taenzer's is still too slow for immediate 


application, but it FHS vfl rthe 1 es J tackles the problem! of practical 
infection. 

Another advance in doing applied vision was reported in Lozano-Perez' 
work on a system celled "PRQPAR" which borrows from natural language work in 
the sense that It generalizes the Augmented Transition Network parser idea 
in a direction that makes it a workable way of describing real Jnages- 
LoiAno’s paper shows that the approach vastly simplifies the recognition of 
electronic circuit parts which previously was done by cumbersome, opaque, 
and less powerful programs, 

Motion is another area of interest Since aany applications in both 
manufacturing and recDtE navigation require a tracking capability. Speckart 
has put together a System that tracks a bouncing ping pong ball. He is 
working on Improving both the image handling and the common sense 
understanding of this system so that it can handle general images with high 
accelerations* 

Manipulation has continued to progress as well* Raibert, for example, 
has succeeded In writing a program by which an am learns about the dynamics 
of trajectories„ Because of recently solved hardware problems, his work to 
date has been confined to three Joints, and more work is planned- 

Work on force feedback has gone forward using the older, simpler arm by 
Sliver, now improved by the addition of e tilting, force sensitive vice. A 
radial bearing has been assembled automatically. In the last few months 
this was cnmplenerted by insertion oT resistors and 1C dips into printed 
circuit cards, (To be sure, special purpose machines can da this, but At 
great expense., Moreover, we have dape this exercise primarily because 
issues are addressed which generalize and lead toward assembly problems 
beyond existing special purpose devices, ) 


ieurjfijip 4nd Debugging 


Some drugging Issues are addressed by Goldstein's work on a personal 
assistant scheduler. The Idea of debugging Is of central importance, 
despite the apparent difference in the selected domain from others in which 
debugging is studied. Two versions of a scheduling program have been 
developed, experimented with, and compared. The first was a simple LISP- 
based, brute force program with an failure and little debugging; it was 
implemented to serve as a point of comparison. The second version was 
implemented in CtilfNTVEft and involved backup along with procedural 
representations of people and their preferences. The third version, of 
Which many modules are complete, is based on the more advanced frames 
concept for representing knowledge and involves ongoing Implementation of 
Goldstein's ideas about debugging and bargaining. The version now under 
study is to work in part by simple first pass approximations followed by- 
sophisticated conflict-resolving tfebngginf specialists, ^e believe this 
could lead to a much more accurate and knowledgeable theory of project 
management Since the problem of scheduling in the face of diverse demands 
and tradeoffs U common to all hard scheduling problems. 


Systems Wdr* 


W® have argued at length that the era of time sharing is eventually going to 
Wrf because of startling reductions in C.PU and central memory costs, to 
hasten this development, with its corollary reduction in system complexity 
and software costs, we are working on a so-called LISP machine. Now $9* of 
the logic design for the ■ draft M version of the machine is Complete and it 
wili be sent out the door for Wire wrapping of the boards imminently, 
Ccrn si derail a code has been written as well. 

To be precise, the system will work by way of two levels or 


interpretation: LISP expressions will first be compiled into 16 bit* high, 
level "macrocode-" The macrocode will than bs translated into the LISP 
machine’s particular microcode. Host of the first pass L.I&P to macrocode 
compiler is now complete. About 3flk of the macrocode! to microcode compiler 
is also done. One version of a complete systen is running with, sioolation 
of the two compilers running on the PdPi0 using PDP1B instructions as the 
target language instead of the LISP machine's microcode. 

During the last six months other system wort has included the 
installation of £5Gk of hew core memory on our PD?10. As expected, dramatic 
iaprovamant in system performance has resulted and -people are once again 
working with and learning from large programs. Until this memory was 
installed,, really large programs ran so slowly they were beyond practical 
debugging, 

Intelifjfent remind is dn-d Very terpe Pcrte flases 

Our work on these programs started in January of this yoar and our progras$ 
is still largely in the form Of proposals rather then accomplishments. 
Nevertheless, we have invented and plan to work on several notions we 
believe to be of fundamental importance to work on the large data base 
problems, for example, the ideas of SGltantlC paging, semantic cache memory, 
semantic interleaving -of execution and fetching, and the use of the semantic 
difference net idea for searching. 


APPLICATIONS 

ORIENTED 

BASIC 

STUDIES 


applications oriented basic studies 


The Artificial Intelligence laboratory proposes to work oh issues central to 
the large data has# problem, the intelligent terminal problom, and the 
natural language engineering prohlea with a broad research projjrass based on 
the following tanots: 

The key to making computers more helpful ahd Affective lies 
in how we represent know ledge In them, tlfrw representation 
ideas have exciting prospects, Natural language interface 
engineering is important. Thera is a deep hied to study 
common sense reasoning. And people must be understood as 
well as the files, 

the large data babe problem 

It U now possible,, indeed common, to record huge amounts of information. 
Unfortunately there is no companion ability to- use that information 
intelligently, Large files have become black boles for knowledge in rainy 
cases, military equipment aaintenanc# records being a particular example. 

Getting useful inf ormatlosi out of large files is thus an important 
practical problem, It raises important theoretical issues as well. 
Artificial Intelligence has done well in tightly constrained donains — 
Hinourad, for example, astonished everyone with the expertise of his blocks- 
world natural language system- Extending this kind or ability to larger 
worlds has not proved straightforward, however. Even dealing with a story 
in an elementary reader seems to require a great deal of background 
knowledge and more sophistication of represen Cation Handling large 
technical data bases may bs harder still. The time has tone to treat the 
problems involved as central issues, 


TAo Largs Pete East Problem Arad The ffliel(ipent irapJ Probierar Overlap 

The Celiac ticn of ideas wh propose nay be allocated among several headings' 
large data bases, intelligent terminals, and fundamental artificial 
Intelligence research. We stress, heaver, that there is considerable 
overlap both with respect to needs and tg patent tel solutions. To be Sure, 
we feel that knowledge representation and organization Fern the core of the 
large data base problem while natural language engineering, comon sense 
reasoning, and user models fprm the core of the intelligent terminal 
prohlem. But we also believe that neither a large data base program nor an 
intelligent terminal pregrun can be fully successful without substantial 
dependence on existing and needed artificial intelligence work in both 
areas . For the moment we dwell on large data bases and on how something 
like an Intelligent terminal system might be used in concert with large data 
bases. Later on fundamental research is discussed la ISSUES QfilEHTEQ BASIC 
STUDIES 

Tfte SmuU Pate ffase ProJHefl Suggests frfeus 

Before talking about what Is difficult, let us begin with what is easy. 
Some or what we would like to dp with large data bases and intelligent 
terminals has been illustrated already on vary simple domains Involving only 
small data bases, The work is important because it serves to establish a 
context for the discussion of large files. A glance at the hotter dialogues 
With 'Winograd p 5 now classic natural language system shows several Important 
Capabilities already demonstrated in miniature. 


i TAp system Aos some uaderi tcfldlitp oj natural language This 
ii essential, not'merely desirable for the largt file 
program because establishing an enthusiastic conmuhity or 
users can be easily prevented by just a little top ranch 
complexity. 

• The system nrateftea /or prrrticiiiffr events St stands ready to 
alert the user to disturbing corbLnstior.s. Constantly 
active •demon" programs watch the flow of information into 
and out of the data base and complain if appropriate. 

■ The system con tfltrospeel iflto some 0/ its sent aciiiri lies j 
tt c<w answer questions about Its own u"u.! structure This is 
Important because we will Often want a large data base 
system to explain how it arrived at a conclusion or to 
supply additional arguments to the ones already given - 

tt The system ira^sEates tap E is Pi deJ cr ipt iafli into search 
programs This is an important precedent beeease we believe 
data bases should eventually include English -descriptions of 
themselves end how they can bo accessed and used. If these 
descriptions can be understood and exhibited, the problem er 
dealing with diverse data bases can be considerably reduced. 

The English understanding required, being Specialized to 
data bases, may not be much more difficult than that 
encountered in the blacks world! 

A natural question is How nuch distance lies between these talents of 
Win□grad 1 s system and something that would be useful to 4 doctor, 
congressporson, or economist? We believe the distance is considerable In 


feme dimensions thd siiQht in others. In the next action 
dimensions seem critical to US and where we Alievs 
allocation Is required, 

IDEAS AND SOLUTIONS 

JVcuf Jjt flfprejenti jIJ ffnoudsrffle Kave ftcEtfnff Prospects 

questions must be answered by finding relevant facts. Finding and re levant 
era the Kay words. 

yn. believi that coping with the problems of accessibility atld rele,ancy 
in largB data bases requires understanding of those large data bases In 
terms Of hierarchically organized frames. This Is crucial. Creating and 
using hierarchy in knowledge representations is difficult, and the problem 
has come to be widely recognized as a central dominant thene firmly entwined 
with nearly all important artificial intelligence challenges. 

Minsky's frame theory has treated considerable excitement precisely 
because it addressas the representation problem squarely and. provocatively* 
Here Is the essence of the theory, as explained by Minsky; 

Vhen one encounters a new situation (or aakes a substantial change 
in one's View of a problem), ono selects from memory a structure 
called a frame . This is a remembered framework to be adapted to 
fit reality by changing details as necessary. 

ft frame Is a date-structure for representing t stereotyped 
situation like being in a certain kind of living room or going to 
a child's birthday party. Attached to each frame are several 
kinds of information, Some of this information is about how to 
use the Trane. Sana is about what one can expect to happen next. 

Some IS about what to do If these expectations are not confirmed. 


■ne exp lain which 
heavy resource 






^0 can think of a frame 45 4 network oT nodes and relations. 
The “top levels" of a frame are fixed, and represent things that 
®ra always true about tfoa supposed situation. The lower levels 
have many terminals -- "slots” that must be filled toy specific 
Instances or data. |aeh terminal can specify conditions its 
assignments must neat. {The assignments themselves are usually 
smaller "sub-framas . n ) Simple conditions lire Specified toy markers 
that might require a terminal assignment to toe a person, an object 
of sufficient value, or a pointer to a sub-frame or a certain 
type, ftore complex conditions can specify relations among the 
things assigned to several terminals. 

Collections of related frane-S are linked together into frame - 
systems . The effects of important actions are mirrored by 
transformations between the frunes of a system- These are used to 
make certain kinds of calculations economical, tp represent 
changes of emphasis and attention, and to account for the 
effectiveness of "imagery." 

For visual scene analysis, the different francs of a system 
describe th(j scone from different viewpoints, and the 
transformations between one frame and another represent the 
offsets of moving rrora place tp place. For non-visual kinds of 
francs, the differences between the frames of a system Can 
roprosernt actions, cause-effect relations, or c hanges in 
conceptual viewpoint, Dif ferent Frames oF a system share the same 
terminals ;; this is the critical point that nukes It possible to 
coordinate information gathered from different viewpoints. 

Much of the phenomenological power of the theory hinges on 
the inclusion of expectations and other kinds of presumptions, A 
frare 1 £ terminals are normally already filled with * default * 


assignments , Thus, a frame may contain a great many details whose 





















supposition is not specifically warranted by the illation. Thtw 
have many uses in representing general informal;ion, most likely 
cases, teshninjUM for by-passing "logic/ and ways to make USlful 
fi&nfrralizatiom* 

Th 0 default issisnffiiifits are attached loosely to their 
terminals, so that they can he easily displaced by new iteas that 
better fit the current situation. They thus can serve also as 
■variables" or as special cases for 'reasoning by example/ or as 
'textbook cases/ and often make the usb of logical quantifiers 
unnecessary■ 

This, frame-systems are linked, in turn, by an information 
retrieval network- When a proposed frame cannot be made to fit 
reality — when we cannot find terminal assignments that suitably 
natch its terminal marker conditions - this network provides a 
replacement frame. These inter-fraae structures make possible 
other ways to represent knowledge about facts, analogies, and 
Other information useful in understanding. 

This theory Is a scholarly integration of some old ideas and many now 
ones. It opens the way to research on previously intractable questions, Tn 
particular the problem of establishing a correct context Is exposed as a 
problem to work on rather than a problem to worry about, nicisky integrates 
and elaborates several approaches in considerable detail; 

1 The Idea of clusters of frames organized like peaces on a 
road map around dense local paths, larger and lfrii dense 
arteries, and finally the limited access superhighways, 

■ The Idea Of group chdracleritation by multiple prototypes. 


d-capitols as he calls them. 





■ The Ida* of moving from -one hypothesized frame to soother by 
way of difference analysis, the so called similarity net 
idea. 

I The Idea of preserving as much information as possible in 
moving from frame to frame by krawing invariance properties 
of terminals and subrrames. 

Elaboration of these is important to the development of entwined 
representations citable of enabling fast access to large data bases. 
F&hlman and Grossman, are wording op other nechandswhich may wall prove to 
be important prerequisites. They arc motivated by a foelmg that 
intelligent systems should have a very fast, perhaps even parallel mechanism 
for working with semantic networks. They argue that questions like these 
should have nearly instant an suets; 


Which ships have bad boilers? 

Are elephants long nosed marsupials? 

[to penguins fly? 

F&hlman "s progress report efferes a solution in the form of a theory of 
nodes, links, and propagating lahels, The theory was developed after 
Fahlmar. tried to apply frames to large problems and stumbled Into a 
fundamental set of problems; symbol mapping (the problem of finding the 
properties of an object Whose description is in the form Of a tangled, 
multi-layered fraae); inconsistent fact detection and the enforcement of 
restrictions; and of finding the right frame during recognition. FaJhnon’s 
hardware nets, described in an appendix, are one possible solution to these 
problems which eerieinly rust be solved before frames can be applied to 


large data bases.. As yet the ideas are new td us and we have- ni j solid 
feeling far whether a Fahlman machine would be a break-through or a 
disapp&intBBiit on the order of the resolution-based theoren proven of the 
past, or the purceptron-based 'learning machines." It is clearly important 
to find out. 

Wo believe that all Of she frame accessing ideas will be increasingly 
necessary (and effective} with Increases In data base size. As yet, 
however, the frame Idea is largely untested. flany examples have been, 
suggested to Illustrate how frame Ideas night be applied, but no working 
system yet demonstrates their real strength, 

We have some evidence that frames can be used to structure large 
amounts of scenario knowledge of the sort involved in children's stories. 
Creating such a system for a. problem of considerable real world importance 
Is a high priority objective. We have not yet devised frame structures that 
would capture such scenarios as nations In conflict. We imagine that them 
could be q Trane theory for the participants in international conflict 
scenarios that would havg terminals and defaults for friends, enemies, 
attitudes,, traditions, desires, wealth, and occupations. (Abelson has 
drafted interesting caricatures of such representations, and leaves open the 
problem of rilling them out with "large data base" detail.) Such a line of 
work would hopefully lead to systems capable of hypothesizing conflict 
frames early enough to activate the mechanisms that could help ovoid their 
consequences. Having such structures should allow frame inst-antiat ion , for 
example, from the +'ern forJt firaas news summaries,, in the sane way that 
birthday party scenarios could be instantiated by stories from children 1 * 
readers, as suggested in Chamt-ak's work. We hasten to emphasize that 
dealing with something so complicated as the \e >V YtrH Ti^fJ is very hard and 
must be considered long range. Vet some of the problems of representing the 
desires, moods, and likely actions of pEople and countries are suitable for 


direct work now. 


Frairce Structure; Sujjtst ftrsurn ijfft fofl Schemes 


Ira Goldstein lias pointed nut that frane theory leads to the nottoh of 
Semantic each ft nemo ry . Selecting a frame by hypothesis, bringing that I'rume 
ihtc fast memory, and leaving it there until dislodged by newer frames is. 
good in the sane way that ordinary cache- memory is good. Once referenced, a 
frame ±5 likely to be referenced again soon. 

It Is ogiially true that frames closely related to a frame being 
processed are likely to become relevant. Consequently the Idea Of paging 
Comes to Bind, Semantic paging would group together semantically related 
Information units that tend to be accessed tegether and could he used to 
bring possibly relevant information up to at least thn fast secondary 
storage level, 

In addition, we sosjiect til* frame hierarchy, similarity networks, or 
other frame link inn mechanisms should fie used to automatically .firing 
particularly attractive Trames out of semantic payes into the semantic cacho 
while processing proceeds. Thus expectations would fie used in an overlap of 
sjuiantic processing and fetching functions analogous tn the old idea of 
overlapping the executing and fetching activities in ordinary machine 
instruction processing. 

Suppose, for example, that a large date base system Is asked to supply 
information about world oil price Stability, Dealing with this, the system 
would follow a thread from the Oil Trane ta the international meeting frame 
instantiated to specialize It to OPEC gatherings- By virtue of a semantic 
cache, frames encountered along the way would be retained,. By virtue of 
overlap between semantic processing ant) fetching, other frames would be 
brought in dealing with pricing policies, recent geopolitical events, 
environmental, and other related issues. 










flfatvreE Lunguagi} /flier face Ef>gineerii\g fj /mpo rtaji t 


Representatlon of fcn owl edge, as said before , U th& most vital part of our 
propound focus on large data bases, Sut other topics strongly related tq an 
intelligent torninal focui deserve discussion because ia our view they can 
bfl studied most effectively by a synergistic program that deals with both 
largo data bases and Intelligent terainals in a unified way. 

Natural language understanding is an example of a technology that seems 
vital to both efforts, It is fortunate, therefore, that this la coo area 
tdiere we feel hard work will surely produce good results even though some of 
the desired scenarios will require considerably mor# natural language 
strength than did Winograd's blocks world task. There are now many natural 
language systems with many good ideas. The field seems ready to gel; it is 
a tin® for integration. 

It seems clear, incidentally, that natural language should not be 
limited to accepting questions in English -* to be really useful, th-e system 
should arrept questions in English, should generate natural English rotiorks 
and reports, end should be able to estreet information rrom large English 
riles as well as highly structured files. 

In particular we visualize bringing together something like Pratt's 
LINCOL or Mood's Augmented Transition Net grarjiiar or Marcus* new Wait-and- 
see grarnoer {described in an appendix:) on the sentence fragment level, 
Filnore's and Martin's Case analysis on the sentence lflvel, and SrhanV S 
conceptual dependency theory on the semantic frontier. 

An analogy with progress IP Cortputer vision research may help clarify 
why we believe rapid progress can be made toward wrapping up work, th natural 
language research at the level pf natural language syntax and elementary 
sflfliantics. On entering a completely new, unexplored aroa, the first StGJJ 
must be to gat a foellrig for how to describe things, usually making 
simplifying assumptions that place interactions in the background, Cuban's 


early work on analyzing polyhedra 14 a good exuple* 6y talking at many 
samples, he developed an hoc theory of how regions Should be or Duped lt,ta 
bodies^ In this early phase, It was appropriate to focus narrowly -* tho 
way shadows Interact with other sorts of lines was justly avoided. 

During the pent stage of research In the drawing analysis work, 
descriptions were refined. This made the results of analysis tnoro precise, 
hut analysis became more complicated because enrichment means alternatives 
and alternatives often load to ambiguity and subsequent search. It is 
proper and natural In such circumstances to tolarata inefficient seareb¬ 
ar ten ted strategics for using knowledge sd as to axposa. through experiment, 
where the key knowledge problems really ere. Later on, however, is 
descriptive apparatus becomes still richer, the interactions becooe so well 
understood that alternatives can be explored not through classical search 
but rather through deferent of all decisions until choices can be mads with 
complete confidante and total conttltmant. Waltz' work represents this stage 
In line drawing analysis; first lines are described in a rich language 
capable fit noting lighting characteristics as well as a line's physical 
cause ** shadow, crack, boundary or whateversecond, constraints among the 
lines are so well understood that possibilities for individual Hues type 
identification can be carried forward until each untenable choice can be 

logically eliminated , There is never any backup. 

Syntax analysis seems in a haalthy state jest under the final stag-e, 
Here find mare evidence Suggests that the natural ambiguities that occur in 
language should 6 ms handled not by guess and backup but by a wait and decide 
strategy. This in brief Is the thrust of some maturing work by Narcus, an 
advanced graduate student. His progress report on this subject, included ai 
an appendix, has a comfortable feel. TTte ?ar«r described already handles 
the following retber complicated sentences in the sense that it separate* 
them into noun groups and establishes the role or each noun group in the 

case structure. 


Vesturday, Oefl-n gave the bock to the girl. 

Who did John dive all or the backs to? 

Is there- a. hoy sitting an the benchT 
John wanted the boy to be given a beak. 

The bey John wanted! to persuade to sit or the bench hit him. 


The system makes the vital structural distinctions between 

They -are railing recks. 

They are breaking rocks, 

which are not at all Similar semantically. The system also handles these 

Wn believed John. 

John was believed by us, 

It was believed that John was tell. 

Ha believed that John was tali. 

He believed John to have been tell, 

John was bellevod to have been tall. 

The boy John, was believed to have hit gave me a bonk i 

The boy John, was believed to have been hit by gave me a booh., 

Has the man sitting on the bench* 

Has the nvan. sitting on the bench walking to the store yesterday? 

Is the girl John saw sitting on the bench walking to the store? 

Is the girl John saw sitting on the bench? 


People ffuit fie tfnderit-fhod As Veil dj Tfle Fi l ft 

Eventually systems oust bo intelligent not only about the inrarnwl ion they 
Contain but also about the people who ess them. Lite respected assistants, 
they must know the shape or what w§ want fron brier instructions if they are 
to supply us with what really concerns us as well as with what wb 
specifically ask for; they should not trouble us with unreasonable demands 
for describing subject detail or stupid questions about bow much we Already 
know r Only if they understand us to that degree can they get the message to 
Garcia* 

Suppose, for example, a known user steps up to a machine and asks for a 
report Oft turkey. That should be enough, in the sense that good default 
presumptions ained at knowledge about people, should hendlo decisions about 
such questions as these-: 

■ Is the user talking about th& bird or the country? 

■ Where does he like to get information rrom? Tho 
Encyclopaedia? The flew York Times? 

■ How much information does he usually want? Two pages? 

Twenty? Aji abstract? An annotated bibliography? How u 
the quantity decision related to recent evidence of 
Interest? 

ft Presuming the user is talking about the country. In what 
particularly is he likely to be interested 7 Is he normally 
interested In information about 5-ovlet border issues, \ f.s, 
old, heroin production* the Dar-flaiielles* rugs, Cyprus, or 


what? 


If the user asks to be kept informed in the ev«nt of a crisis, then the 
question* of whal be is Interested In Defame:* two dlnensionalt the system 
Bust know not only the topics that are to be monitored but also what 
constitutes a crisis. The fact that sobs U.S. tourist gets Jailed for 
buying dangerous drugs Is not likely to be a crisis to Henry Kissinger but 
it might be for a local embassy official, The Secretary of State presumably 
Hants to hear about how the Turks are likely to react to changes in the U.S. 
aid flow, while the fellow who worries about tourist problems probably need 
not ha awakened each tine new info mat ion on such subjects appears. 

Proper monitoring can he handled by development of a theory of user 
models and a theory of semantic interrupts. For esanplo, such a theory 
Pight Involve a technique by Which interrupts would be generated Whenever 
specific, user dependent frames are activated by tentative hypotheses or 
confirmed by a complete analysis, 

AH of the work on "demons" derived from PLANNER and CONNIVE,R work is 
relevant, of course, but we need in addition an epistemology of people and 
what they know. We believe this is strongly related to the other side of 
the system, the machine, its large data bases, end what they collectively 
know. In particular, we believe it will be possible to Use the same frame 
oriented knowledge representation theory for organising knowledge about both 
people and the domains that interest them. It should be very productive to 
keep this in mind, especially since the ultimate system cannot be had 
Without significant progress on modeling both,, however Sim liar or disparate 
they may be, 

Naturally users should have easy, smooth mechanisms for resetting the 
knowledge the system uses to decide when to bother them, Probably most 
users will require something betweo-n absolute importance thresholding, and 
Constant information rate, independent of how nuch action there is -* 


*■nnethin U Ilk 1 * ^.1 it yet an the finsnnlal pages. 

Sane relevant work Is already Underway to handle the problem of people 
and their preferences in the personal assistant context. After til, even to 
arrange a person's schedule Some facts about when he likes tb get up, whom 
Pie likes to see, Wlt*n he Can tolerate Interruptions, how important ho is 
relativa to various superiors and employees, and the like raist be known and 
used. AS additional modulus are added* particularly in. the dimension of 
automatic note digestion and information retrieval, more progress in tills 
direction will bo needed. 

For e major large file program, however, additional professional help 
seeds Of preeminent importance, In this We are fortunatE to vnrk in clhse 
collaboration with people having deep interest and considerable experience 
in thinking about how people think and what they khow. He can consequently 
propose a broad approach, one dealing both with the artificial intelligence 
issues end some of the human modeling quest Ions.. Indeed wu believe that a 
synergistic approach based on such collaboration is far more likely to bo 
productive than separate, divided assaults. 

Seeer< t[ flomfias -Secrt Suitaiiie for ffre Juste Jesoorcih Wort 

We want to lock at the activities of various professionals including the 
following: 


■ The Equipment 1air.tenar.es Officers. This of course is the 
prototypical large fils User. Direct attacks on existing 
files Bay OlcUd some issues at the start, hut as a domain 
for study it is interesting by virtue of the potential slat 
of the data base, the need for trouble oriented watchdog 
demons , the possibility of using cormon sense qualitative 
reasoning,, ahd the need to maintain sensitivity to user 
suggestions About Just how to watch and monitor things. 

V Thn Strategic Materials blilysists, The idea is to Worry 
effectively about reservos of various metals and other 
strategic materials. The problem features both hard tabular 
data and English phrased opinions about political trends. 

B Tits Logistics Planners, the Vast problen of getting things 
Where they Should he and tracing blunders derserves careful 
attention. Part of the problem Is a resource allocation 
problem with conflicting goals. 

Again we stress our opinion that all of these areas have much in common 
in terns of ths scientific challenges they offer, 

SOME QUESTIONS 


Why 00 b> to ltc*rt t ro t p On fpxt ftiej? 

We do not, really. We believe that the text examples we have giv^n cover 
the main issue? of semantic organUStlOh that all largo data base problems 
present. Certainly considerablo thought will be given to nore structured 
files, as well. We favor selection of target domains in which combinations 


af text and sIPUCElired data naturally occur, An example would be 
maintenance records In which free text and highly constrained Forms are 
blended toga Char. 

Wow iefge Is Large? 

We believe the hierarchically structured, frame oriented approach we propose 
will lead |o systems that perform well on large data bases whatever the 
size. Larger data bases will require more muscle In tho fefiowladgo 
structure, but nut in linear proportion. By adding a layer or two of depth 
to a hierarchically organized! data base, hugs increases in the size of the 
information covered Can be achieved using basically the SBmo theory used at 
higher levels, The main obstacle then becomes that of understanding the 
naterial fast enough to prepare the lowest level description frames os it 
goes flying by. 

At this time 50 little has been done we Cannot realistically speculate 
OH the size of the data bases that will be handled in the near and raiddlo 
term. But since the knowledge organisation research we propose seems 
fundamental, we feel it will be central to handling any data bases that can 
be handled at all, 

THE PROPOSAL AND THE MILESTONES 

It is clear- that work on large files interacts very strongly with other foci 
of laboratory Interest. While it Is not possible or appropriate to draw 
Sharp boundary lines into the picture, it is possible, however, to guide 
both new and existing resources Coward a large file focus, This point of 
view i* reflected In tha following plan, elemehts- Target dates range from 6 
nonthi for creation of a structured flat* base with natural language input to 
30 months for implementation Of semantic paging and similar Ideas. 


1 Implement a structured data base with natural language 
access for a small subset of the relevant statistics for a 
particular strategic material or equipment maintenance data 
base ► This represents a capability already achieved for 
other t smaller domains- Accomplishment of our goal here 
will be marked by being able to direct simple statistical 
queries in English to the data base. 

■ Concentrate our natural language effort in. the intersection 
of the problems raised by our personal assistant focus and 
large file issues. After two years time we would aspect 
good natural language front ends to a variety of personal 
assistant and largo data base modules. 

i Push forward simultaneously with the network taxonony wprk 
of Fahlmon, expecting a F'h.D. in June of 1976. this is to 
contain ideas and experimental results defining the possible 
role df a fast, Wall conceived A-KI^S-OF processor In the 
large file program, 

■ Implement a frame system representing interpretive knowledge 
about the data base. This interpretive knowledge consists 
of both common sense facts such as the fact that oil must be 
refined as well as expert knowledge Such as the time and 
costs involved. This milestone represents the application 
of existing theoretical Ideas to an application area. This 
allows queries to go beyond simpifl statistical inquires 
data hases can be asked about trends, possible dangers, etc. 


fl Implencnt a demon capability for real-time monitoring of 
critical situations, [tcmons will monitor both symbolic and 
statistical additions to data bases. We must solve 
technical protv] ems which cone up when demons are activated 
by complex patterns, and wb must solve problems of 
computational efficiency needed when multiple demons are 
activated* 

V Extend the natural language capability so that a system Can 
add statistics to a data base by processing text. The 
system ip freed from the time consuming and error prone 
process of huran reading of textual data and human entry of 
the relevant facts* Furthermore facts can be added more 
quickly and the Information flow can have less Lir-e lag. 
This of course Is a critical issue for data bases 
representing estimates of the supply of critical defense 
Items* This milestone will be achieved when a system reads 
news articles on a Sanple nateriai* 

■ Use of frame organization and especially hierarchies tmd 
differential Tramo pointers to allow semantic paging, 
Semantic hashing. Semantic pattern matching, 9t is this 
organisation which goes beyond traditional ideas that we 
expect to allow reasonable computation times even on very 
large data bases, 


■ Keep these diverse research activities coordinated and 
integratfl the results Into a pilot system as rapidly as the 
ideas reach a sufficient stage of maturity. This would he 
the duty of staff researchers commencing work in early 1976. 

The result is to be a well documented system continuously 
available for experimental use as several of the natural 
Language systems are now. 

HARDWARE 

It is ■conceivable that a machine inspired by Fahlman's research cm knowledge 
structure might become part of a successful approach to certain aspects of 
the large knowledge base problem. In our opinion, the Fahlman concept Is 
the first hardware design that promises to handle information problem for 
Which the classic Serial Von Neuman architecture may ultimately prove 
inadequate as the t-otai. foundation an which to build a truly intelligent 
system. Wo do ftot propose to begin the design of such a machine Itaw because 
the idea is not sufficiently mature ansi it is not sufficiently substantiated 
by simulation. After this work is dope, however, we hope it will prove that 
the design of a machine should be in a future proposal. 

Meanwhile it la clear that developments in artificial intelligence 
problem areas will require very hoavy demands on confuting resources. 
Experience shows that very large address spaces amd dedicated central 
processors nru required to Cope with the complexity barrier. At the moment 
the Lisp machine concept, proposed for development in a separata section, 
seams well suited to this purpose. 


TECHNOLOGY TRANSFER 


jWc . Pl an to wopX With a vitK toward providing implemented nodules 'with both 
the enchantment appeal and the inherent usefulness required for export, We 
further plan «ariy involvement of people from u selected user community, and 
we railizB that the availability of such involved users is. an important 
con sld.e ration, in dona in selection - 

PERSONNEL 

At least the follo^in-g people would contribute to a large 
Negotiations are underway to recruit other artificial 
specialists from outside tls-e N,I,T« conciunity. 

B Professor [R.A GQLDSTEIh, will be the project Coordinator, iie 
has been loading the laboratory’s personal assistant work 
aijtied toward understand lag the scientific questions involved 
in the development af a system capable of scheduling 
appointments and events, organizing notes and observations, 
and retrieving a wide variety of knowledge, FHID (tEAN Is 
currently helping Goldstein with the scheduler module. 

> Professor nARVlN KlMSKf Hill work on the problems of 
tail owl edge representation, and comnnn sense reasoning, It is 
expected that Kinsky's frame theory will be of considerable 
importance. In studying the problems he will be assisted by 
several praduate Stu deats l 


file effort. 
Intelligence 


■ SCOTT FAHLHAN and RECK GRQS5RAV bath came in depended Ely to 
the view that aome better approach tg retrieving knowledge 
Is denanded by the current problems facing artificial 
intelligence. They ere working hard on the new network 
label propagation idea te handle the sorts of scenarios we 
see Coning dp in the large file context. 

I Professor SLVnoUR PAPER! is interested in the question of 
organisations,, the manager* that exist within then, end the 
problem or supplying these mongers with in forme t ion that is 
at once useful and desired. Such questions wili certainly 
be important in the intelligent terminal context since one 
function of an intelligent terninal would be to facilitate 
the interaction of an entire office full of people. STEVE 
HaSE.All5£fHj and din STAWSFIELb will assist Papert in helping 
to identify the genuine management information issue* and 
use these in the process of selecting a target domain with 
which to work. 

■ Profnssor VALPGUAN PfLATT lind his Students are moving forward 
rapidly with the development of Uk'GdL, a very efficient 
natural language systec facilitating grarmar writing. This 
Is expected to compete favorably with Wood's Augmented 
Transition Network formalism for the prize- of being the best 
sfpple system for implementing natural language front ends. 


■ CANDACE BULLU]NfcLE, DAVID MCDONALD. and HITCH MARCUS are 
graduate Students Specialising in natural language studies* 
Btiliwinh; le Is working on a module far interfacing various 
personal assistant Facilities to users , McDonald is wording 
on knowledge based dialog, and Marcus is finishing a 
powerful new parser believed tn integrate the best features 
&r a number of separately developed systems. All together 
Wa See this work saving toward an understanding of the 
roiatlonship between task si^fl arcd complexity on one Hand 
and system sophistication, on the other, Natural language 
interface engineering seems plausible now. 

APPENDICES 

FAHLCTAAf: A SYSTEM FOR REPRESENTING M3 USING REAL HQRLD KNOWLEDGE 


NARCLS; WAIT-AND-SEE STRATEGIES FOR PARSING NATURAL LANGUAGE 


applications 

ORIENTED 

VISION 

AND 

manipulation 


APPLICATIONS ORIENTED VISION AND MANIPULATION 


Th& Artificial Intelligence Laboratory proposes to work on applications of 
computer vision and manipulation with the emphasis shifting from tool 
building toward, solvinij real problems,, using we 11 -known, comnercislly 
Important problems as a source of challenges, In particular, we plan to 
devote part o-f our effort to e hard assembly problem Involving putting 
together or taking Apart devices with flexible components, awkward 
geometries, and poorly constrained tolerances. We further plan to broaden 
our activities by inclusion of work on tracking moving objects end on real- 
world inspection, these problems now being within reach of solution• 
Success on these new problems will lead naturally and smoothly to progress 
on some aspects of the general problem or applied image understanding, 
although not all of them as wq argue later when we discuss the future of 
basic studies in vision, 

IVcHfucttntttf Technology flsj /Iff Jmperttfrtt fUsiiM 

Broadly speaking, the overall goal of pur work in applied vision and' 
manipulation is to develop uses for artificial intelligence techniques in 
hand or within reach now. We have this goal because we want to do the 
following things; 

■ Haintain the economic strength of the country in the 
competitive international irnns through automated 
production, particularly in assembly. 


■ Create a more flexible brand pf manufacturing capable of 
vary rapid response in times of rapidly changing need. 


■ Prepare for automated mining, Doth Underground and undersea, 

US wall AS automated ferning, space exploration and 
racovery, ordinary maintenance, and maintenance in hazardous 
anvironneEits. in part this means creating an autonomous 
VehicaJ guidance technology capable oT coping With 
unprepared environments. 

Fa ieiteoe Sfieae ofi-jectiyes ore ^iEoHy importorri Aecunise a p reenrt n p J] t 
computer science and trdestrp is ojie 0/ the mo for street to; 0/ the ffntte-tf 
St-fftes. Ew'e must tfdiuktt ihu a lie en to ge i ra t. 0 corresponding cipuiL E; ties in 
ereaa where ppr fleiiliiofl is erotfinp, especieEI]f In producttotttf, Wij 

think changes will come about quickly when a certain threshold of technology 
is passed over. Just as pocket calculators swept the country like a tidal 
wave, wo Pulieve computer vis inn and manipulation will soon sweep through 
the manufacturing and maintenance industries, Hut before discussing what 
a non means* let ns look at thE path on which w* have been on and how it 
reaches the stated objectives. 

fropress Jit fro ns/erring ^rti/icloE InteU iflfl.nee Ideas To Industrial 
fTactics Iavofoes four flejpr 5tW 

We mo a sum progress in terns of where we are in executing the following 
plan; 

1 Pick the si op lest possible domain* do vision and 
Manipulation in that domain, and use the result; to measure 
the difficulty of the problems in order to allocate 
resources and predict when applied results should emerge. 


This has be eh done, uf course- The blocks world was tho dona in; the 


difficulty of vision dictated the concentration or effort on it and the 
relative neglect of nan ipu let ion. It becaae clear that the existence of 
global knowledge and the flOIl"linearity of noise effects render cost previous 
work on image processing of little value, Instead, problem salving, 
constraint exploitation, arid symbolic low level processing became central 
Issues as discussed elsewhere in our discussion of directions for basic 
studios iti Vision t It also became Clear that vision is Very hard, and for 
tbe sake of bear term application results, the peals- in vision research 
should be divided into no-tricks basic study and no-hoIds-barred. special- 
purpose, domain-do pun dent work, 

2 Once thtt problems are uncovered through suffering ip the 
trenches with the "simile* dona in, then it is time to get 
together hardware and software tools capable of tteaningful, 
cost conscious prototype arranggjiBiit to solve particular 
problems, 

This too is behind us, Our so-called Hlcroautsmaeiou Laboratory exists 
and supports applications work, It is described in brief later on, 

3 Use the tools to techie real problems. At first thesE may 
be selected because they are amenable to solution through 
the tools and techniques in hand -- obvious undlsputable 
commercial viability is a secondary objective. 

Our work on electronic assembly was chosen for suitability in this 
phase. Many results, described later* are in hand* Ug believe we will 
COBplite the remain lug work in electronics this year. 


4 Finally, uni bus h the technology op the most sBrians problems 
whore the saving would be greatest in human and fiscal 
tonus, 


This is to bo our primary purpose during the contract period, At the 
moment, this purpose translates to th& assembly of mechanical devices 
exhibiting all the tough asSGQbly problems we cart find which seem essential 
to industry and apropriate for solution by some cotriblhatioti of vision, force 
sensing manipulation, and now understood Artificial Intelligent# programming 
ideas, 

* Industrial exposure is of course vital to a program like 
this where sensible resource allocation and technology 
transfer arc so important, V& are therefore fortunate at 
BIT to have a vary active Industrial Liason Office which has 
helped arrange for us a wide spectrum of contacts in 
industry, 

Production facilities visited include those of Texas instruments, Data 
General, Digital Equipment, I0h, the RCA hybrid circuit facility, the 
fcecinBcott Copper Company Peabody coal mine, General Rotors car assembly 
plants, Dclco integrated circuit facilities, ARP incorporated electrical 
connection and electrical assembly facilities, the Foxborn industrial 
controls facility. United Shoe Company's autopated electronic equipment 
■tchlnery production, and the GCA facility charged with the production of 
advanced lead bonding equipment, Bore of this industrial contact is 
planned, in particular a number of advanced production facilities and 
research laboratories will be visited in Europe this fall, again to be 
financed by the MIT Industrial Li as on Office. 

Vb will return to the subject of hard problems in a moment. 


THE FOUNDATION FOB THE FUTURE 


Ve now give a brief description of the Laboratory, move then to a brief 
discussion of some progress made wlch it, and then finish with, the proposals 
Tor the future- 

Ir our Last major proposal, we onphaslzed two opinionsr 

V Up argued a need for a workable. Inexpensive set of hardware 
and software tools far doing research in applications 
oriented Computer Vision end manipulation. We laid out a 
plan for the creation of these tools- 

1 Ve suggested that tool creation without one or two target 
domains can Ot inefficient end leads to pour resource 
allocation. Wo specified tiie problem of electronic circuit 
card manufacture and repair as suitable for bringing general, 
issues into crisp, manageable form. 

All of this work is well along as evidenced by our progress report for 197$.. 
The Inexpensive, exportable vision and manipulation laboratory now exists.. 
It has demonstrated its usefulness by supporting important target domain 
projects involving electronic, circuit cards and by facilitating first steps 
outside electronics in Other worlds Of interest like the world of small 


mechanical devices. 


rfrff Hardm re Is fttadff 


We have a PDF 11/40 with extended instruction set (EES}, 51ft of tore mEmory, 
two 1.2 meg a word disk dfivaa, and a GT40 graphics display terminal. In 
addition,, we have the following devicesr 

■ Two VlCARPf mechanical arms, 3/4 human size, with six joints 
and si* degrees of freedom, position and velocity sensors In 
most joints, and gripper hands. These emprise our primary 
manipulation system. Tesas Instruments, the University of 
Illinois, and the Naval Research Laboratory have ordered the 
HIT designed arm and wore are expected to follow suit. 

Figure 1 shows tho general appearance of these arms, 

■ A Vi(Ucon picture digitizer, consisting of a television 
camera and a television monitor, and a digitizer which 
converts tho scanned Picture to £ hit digital data at a 
maxitiium rate of §5- microseconds for each point in the TV 
scan. This is currently our primary vision system, 

■ A Huijhes modixlatable laser With an output of ,061 Watt, used 
principally as a pointer or tracking davits. 









i Twg motor-driven pirror-deneetion systems, by Spatial Data 
Systems, Inc, Each mirror system has tvo independently 
controllable mirrors with perpendicular ax-is of rotation, 
and settling times of approximately 3 milliseconds for 
motion of the mirrors. One systEn is used to deflect the 
Hughes laser; the other is attached to a PIN diode via an 
optical system and is used for scanning scenes as a slow hot 
very linear,, high resolution vision Input device. 

■ An Optronics International, Inc 4 phot;writer, capable of 
making extremely high resolution photographic negatives on 
film from stored digital pictures, 


■ A precision X-V table, made by the Icon Company, capable of 
motions in two orthogonal directions with an accuracy of 
,001 Inch, over a range of motion of 6 inches with a maximum 
velocity of 4 inches/second. This device is used to 
position work under tha an chapical arm or the vision system. 

B An Analogic digital-to->nalosj and analog-to-digitil 
convertor with (3 S to A channels and 64 h to D channels, 
capable of performing data conversions in 3 microseconds, 
and to which we have added an additional 16 D to A channels. 

fA? So/tbjorc fs Seedy 

The Productivity Technology Group PDPl 1/43 uses standard DEC software, the 
Disk Operating System (EW5}, in an effort to make software developed within 
the group useable elsewhere. The group has developed the following 
software; 


■ VIDIW is the visual input program, It is Used in 
COnjunction with either the Vidinan or the Pltf-dioda/mirror 
system to obtain digitized pictures which are stored on 
dish, The User can select arbitrary sub-sections of the 
entire picture being scanned, ar.d the program continually 
displays to him the sub-section of the picture which he has 
selected. 

■ MAPPER, is a prograa for computing statistics on stored 
digitized pictures. MAP?IP; use provide histograms and 
intensity gradients, printouts of the raw data and even 
-graphic representations of the picture itself. Operating in 
cnnjuctinn with a program that funs dn the BT4Q graphic 
display terminal. It can threshhold a picture Into 8 
intensity gradations, and then Cause the picture to be 
displayed on the GT40, rtAPPtR enables the user to examine 
small subsections of the entire digitized picture, for 
detail, or to combine subsections together to get a larger 
view of the statistical analysis of the picture, 

■ LISP, a fulLy-genoral LISP- interpreter, with shallow 
bindings and a non - recursive garbage collector which 
conserves memory, This LISP does have the restriction that 
atom print-nanes must bo 3 characters long, or less. It 
does standard DOS file I/O, and can additionally do input nr 
output from a variety of terminal devices, or from the 
Artificial intelligence Lab's P0P1D timesharing system. 


■ RUG, ft symbolic, DCG-oriented debugger, which replaces the 
POS debugger ODT. Using RLG. a user may debug standard DCS 
format programs- the debugger makes available to it La all 
hit synbals, disassembles the contents of core into assembly 
language statements with symbolic references included, and 
allows hid to enter assembly language code Into core by 
merely typing It in, RUG has various numeric and special' 
purpose type-in and typt-Dut routines (such as ASCII end 
RADIK-50). It allows the user to Set breakpoints In his 
program, and additionally allows him to monitor the contents 
of a core location,, register, or device for some desired 
condition, 

■ A set of arm control routines for the YTCMfl mechanical am. 

These «hP implemented in LIE?, and include routines for 
moving the am to a particular state-vector (essentially a 
position and velocity for each joint), and for halting the 
arm's motion and holding it still. The routines incorporate 
a trajectory planner which decides an the route the arm will 
take to get to the desired state-vector, and a dynamic 
routine. The dynamic routine looks at the desired State- 
vector as cor. put fid by thfi trajectory planner at the end of 
the next time increment, and computes the torques which must 
be applied to each joint in order to achieve that state- 
vector, 

finally, there is an improved editor which Utilises the graphic 
capabilities Of the GT40 terminal to luplenEnt a real-time edit mode, and 
there ere programs for trensmiting files to and from the timesharing system, 
for displaying images on. our G“dO tormininel, end for other typical system 


activities. 


Artificial Intelligence Jieie(j,rch Fee<fi Productivity refihftoiopjr 0evMopweraf. 

The flaw of haste Artificial IiUelIig-Br,ee techniques to applied problems has 
begun in the Artificial intelligence laboratories and a complete list of 
achievements in this direction by the Cdddiunity would surely include the 
impressive work done in -Japan on A. I. inspired ZIP code readers and circuit 
card! inspectors, the work at Stanford on manipulator controls and * ur 
wort, of which the follow log are of particular relevance: 

A InouG and- Silver demonstrated the automatic assembly of a 
radial bearing with 12 micrometer tolerances* thereby 
demonstrating that co.^piiccrletf aas-raiiiteo c-a^i be made us Cap 
/testae* pro ff nrrnmerf rjirstpuimtorJ, The achievement further 
demonstrated that knowledge b lt■:• c t /erce sensitive 
non ipc l o tors ore the try to irdirartCCd vsitatbly operations , 
and that high level pro3 rarrni r-2 larttfkatfOJ mote prog roravtng 
for raerflajTicoZ assembly a Viable concept. The bearing is 
illustrated In figure 2. 

V Horn Showed ftOW tP apply the technique Df Anomiedpe flosed 
profile analysis, borrowed from earlier blocks world work by 
Binfurd, Shifal Pnd Others, to the applied problem of 
orienting IC chips in preparation for lead bonding. 

Figure 3 shows some typical profiles encountered. Hat Pfason 
is doing the same thing with automated diagnosis of 
electronic signals of the sort that technicians lock, at on 
oscilloscopes. 
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■ Taenzfr uses reftoa t?roi:i.nu i n> a j to in&pnct solder Joints 
for the standard defects -- pin holes, cold joints t and open 
holes. See figure 4, 

i &peekurt has begun to incorparat# MpAer Ji^Bi JtflouJerfje 
about obstacles in low level tracking routines that follow 
Objects through OthtrwisB difficult accelerations. Figure 9 
shows a truce of Speckert's points of observation as his 
system tricks & ping-pong ball through a bounce, 

THE CHALLENGES TODAY 

Using our now well developed tools, we plan to accomplish the following 

■ 'he plan to Conclude OUT ARPA sponsored automatic assembly 
program. This will he amplified in a moment, 

1 We plan to deal with a difficult. Important, end costly 
inspection problem, solve it, and use the resulting solution 
to determine if cost effective solution is in sight, W« 
propose the inspection of jot engine castings as the 
can did a te domain. 

■ Wa plan a new program in the dynamic vision of tracking and 
vehicle navigation, We believe this will prove to be a fine 
source of new important problems suitable for replacinj our 
work oft assembly as that phase of our program goes to 
successful completion. 


5? K J x r 


j i i i ■ *; 


!l i L;1 ;j! il h !• I B 





























































































































<""X 


"A 

C/\ / 

^ \\ 


/ 

,/> 

( 

/ 


X 



s 



LOCUS OF POINTS SC/' NG 

a ersu*icf:. P!i:(i pour, hail 


Assenrhfy OJ A (JtfioHflt frtpir&e Address^} The rei/g, 1 ! Issuei 


It seems to us that we should be workins on elements from the overall list 
of problems that are best Fitted for us in terns of our own expertise and in 
tonas of what workers at other institutions are doing wail trough, We 
therefore propose to build the snail gasoline engine *- very likely the 
oadel airplane engine illustrated in figure 6 ** with a view toward solving 
the following problems: 

■ The programing problem, Our intention is that the assembly 
program should look like an annotated parts list With advice 
about the assembly, not low level instructions, We are 
inspired in this effort by complementary work at Stanford 
and by our own history of success with very high level 
languages like PLANNER and COHN IVES, 

■ The parts orienting problem. The parts are to be scattered 
at random on the table or drawn from bins. The SHI results 
will be adopted,. If sufficient. 

■ The flexible part problem. The engine has a gasket, Wi 
hope to conn act up the fuel line as well. We believe our 
handling of flexible wires in our electronics assembly work 
will be valuable experience. 









■ The two-hand coordination problem. The engine has a 

difficult probleu inasmuch as piston ^ piston pin, and 
CSrliaec t in sr rod HUSt be brought top ether. IfiOUU's and 

Silver's work with the radial bearing presented in the 
progress report, gives us some solid close-tolerance 
manipulation experience, 

■ The Straight coordinated movement probleHr In an effort to 
eliminate the complex equations normally required for 
dealing with, arm dynamics. Raibert is constructing a model 
whereby the arm controller learns froa previous attempts at 
pavement control and stores this learned behavior into a 
large data base OP disk. The first Stop is to Cause slow 
motion of the arm between pairs of points in order to teach 
the system the gravitational parameters along the path In 
between. Then the arm cnntrgller can begin to learn about 
the dynantes of ration along the sane- paths by Iterating the 
motions at speedy trying to learn about the arms dynamics in 
order to reduce the gap betweon Eh* previous iteration and 
the desired notion, 

i Coordinated vision and itanlpulation. insertion of the 
crankshaft into the Connecting rod offers this Challenge, 
The try will not win every time, we suspect, bringing in the 
opportunity to deal with fault detection and recovery in a 
natural way. Speokort's tracking work may be useful here, 
ns well as Taeozer 1 s inspection and insertion work, 

V Tools, The entire job will require some serious attention, 
to thought about the sanip-u la tor's tools. 


■ Inspection. We nay, if fortunate, be able to test the 
assembled engine and even adjust the needle valve. 

Inspection for surface defects, however, is to be done In 
the casting domain, described later, 

Costings Seem To ffp rfte Right Jamais For Visi/oE Zmspecti 0 e studies 

The visual inspection pf costing- 1 : is te be e step toward the development of 
practical visual inspection systems. Figure 7 illustrates the sort of Image 
produced by a pinhple defect, one of many types te be studied. In cur 
laboratory, we have been developing the technical tools needled to understand 
images of complex surfaces like these. The current work of Karr, in 
particular, offers insight into what to do because it deals expertly with 
the problem of texture. 

In order to become aware of the practical issues associated with the 
inspection of metal castings, tloodhas has visited the Advanced Casting 
Laboratory of Pratt A Whitney Aircraft and the Draper Division foundry of 
Rockwell International. They tieva been extremely interested and helpful. 
they have been not only a ready source of defective testings but they hove 
Also provided Wcodhaz with tin a da tailed specifications by which various 
parts are currently inspected. These Inspection standards provide the 
Appropriate benchmark for assessing the performance capability of visual 
inspection s ys terns . 


The Dynamic gtsiefl Of Self-Orienting find Tracking I$ fteadg For Study 

Over the years, the problem of seeing and understanding static scenes proved 
£0 hard that anticipated work. On the vision of motion remained out At 
pasture with few important results emerging from the isolated attempts that 
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were made. White there ere still Deny diff iculties , the time has come to 
loot at motion problems seriously because we have better ideas about 
representing visual knowledge, because we hav E LetLer hardware and software 
tools, and because the need plainly exists. 

to this end, we plan two efforts. One now underway as a doc tor el 
thesis project attempts the following scenario: 

■ A vehicle is Dating on or over hilly Kars-like terrain. The 
vehicle wants to (tee? a good model of the ground it Is 
moving over with a view toward tracking its position on a 
map if it has one or toward making a map if it does not. 

The self orienting study will address two key scientific issues; Tirst 
tho issue of dealing with representing situations evolving through tine; 
Second the issue cf exploiting constraints evidenced only by comparison of a 
Sequence Of situations, not by single situations alone. 

As the system looks at the world, it is to sake a symbolic network 
description Of what it Sees consisting of nodes representing hills, hill 
features* and the T-joints that appear whore one hill obscures tho 
background boundary of another hill further behind. The nodes are linked by 
relational pointers with labels like IN-FROffT-ClF, TALLER-THAN, PART-OF + and 
sc on. Figure 0 Illustrates, The changes la these descriptions through 
time* it is believed, will be subject to handling in terms of rtinsky's 
theory of frames, the now maturing theory of network description, defaults,, 
contexts, expectations, symbolic transform, subgrouping, and more, as 
described in applications ORIENTED basic STtMiE 5. Each symbolic description 
of one Instant in time will ha ona frame in a collection. 

As the system composes its frames, it will use constraints devolving 
from motion parallax and from changes in the observed vertices. By SO 
doing, the notion becomes a positive contribution to analysis, rather than a 





SYMBOLIC HETUORK REPRESEUTiHG ABOVE INPUT VIEW 




complication to be removed. Tills aspect of the work hat strong antecedents 
Iti Walti' work on static scene analysis which showed that by proporly 
addressing the constraint question, shadows and the Influence of one vertex 
on its neighbors bacons terrific aids to analysis, not handicaps, 

Another problem, to some extent the inverse of self orienting, is that 
of tracking coving objects: 

1 Constrained objects are moving thruU$jh an image. Knowledge 
of the constraint is to be used to 3i.elp track then through 
situations Involving temporary interruptions of vision, as 
when one abject obscures another; and involving fast 
changes, as when a ping pong ball bounces off of a paddle; 
end Involving objects constrained to move together, as when 
vehicles follow o m another. 

The study In tracking will build on the work of Speckert appearing in 
the progress report. Speckert has tracked ping-pong balls through bounces 
which would baffle most tracking algorithms by combining simple world 
knowledge with ordinary adaptive mechanisms. More of this will lend itself 
to trackers goad enough to, say, track people or to allow vehicles to follow 
One another in convoy, as in a mine or on liars. 

TECHNOLOGY TRANSFER 

We plan several steps which we believe will considerably shorten the time it 
would otherwise taka to bring our vision and manipulation work into practice 
on the factory Floor, The work seqas too important to depend on the slow 
osmosis that would occur without deliberate planning. For the most part, 
these transfer plans involve spinoffs from our norcal academic duties which 
are done with no cost to our rasaarch contract. 


■ We h n v fi Introduced & n ex nir ccu:se on the subject of 
computer vision and nanipulacion in order to prepare new 
graduates for active leadership in the field. A text has 
been undartaken as a methanisu for encouraging such courses 
elsewhere, 

I We have Introduced a summer session course for the MIT 
industry oriented Sommer progran. There was a favorable 
first-year turnout with engineers and managers from Boeing, 
Honeywell, Mew York t*Lej»hp(je f The U, S, Department of 
Transportation, Westinghouse, Kodak, and Xerox attending. 
General Motors has purchased our Videotape introductory 
Artificial Intelligence course, 

i We plan articles for the trade literature describing the 
laboratory equipment and summarizing what has been done with 
it sc far. This is to complement our normal publications in 
the scholarly journals, 

R We have whflnever possible built the Vision and manipulation 
laboratory out of easily obtainable equipment and we have 
avoided modifying this equipment. When new equipment has 
had to be designed, we have avoided OMHof-*-kmd in-house 
construction, 

PERSONNEL 

■ Vision and manipulation work will be done under the general 
direction uf Professor &IH.TU&L& K, P, NORM, well known for 


hIs work. on machine vision, including work on shape Cron 
Shading and reflectivity analysis. 

1 NEVER B1LL HI?, .5 trill be in charge of system software 
development. JQrih PURBRICK Hill do mechanical design. 

■ DAVID SILVER and TD1AS LOZMO-PEHEZ are working with DAVID 
TAENZER, and HARK RAIB-ERI toward ichieveoent of the engine 
assembly. 

1 BOB t'ODDHAK will work on the visual Inspection of certain 
castings. NME LAVIk will do the dynamic vision of self- 
orientation. GLEN SRECAERT will conjrlenent that work with 
his knowledge-based tracking effort. 


ISSUES 

ORIENTED 

BASIC 

STUDIES 


ISSUES ORIENTED BASIC STUDIES 


The Artificial Intelligence Laboratory proposes to e&ntinua a basic studies 
program aimvd at da sling with deep, central Issues. The Issues ara to bo 
exposed by W^rk in computer vision, COrMsoji sense reasoning, and expert 
problem solving. Hie ubiquitous re pres an tat i on problem is Involved in ail 
three areas. 

THE NEED FOR BASIC STUDIES 

All research lias on a spectrum batiueen pure development and pure research. 
Most work, including all of ours, lias far from either gf these extremes. 
Still, one can, either select a need and bring tools to boar on it or one can 
select issues and choose the best domains in which to expose them, the work 
described, in this section lies pore toward the issues oriented focus, where 
it is harder to specify timetables or even to predict accurately what will 
happen. Historically some of our laboratory's greatest contributions have 
cnee out of work like this that can be aimed but not tightly steered. But 
this is not to say that our basic research Is done by guess and lurch, Our 
long range goals are clear. They are described in sections devoted to 
machine vision, COniaon Sense reasoning, and expert problem solving. 

All three of these sections emphasize the importance of sound 
representations, ^fe ficlfeinf that proper reprejeJitotioh is tfte Jfceir to 
fftfuunced otsioa. common sease reersonfn-tf, end expert problem solving, /ust os 
It Is to mpjiir otfler a j peels */ Artificial Intelligence. Indeed a subtitle to 
this section, ISSUES QBEI£)tT£& BASIC STUDIES, could be, ASPECTS OF THE 
REPRESENTATION PROBLEM, 


SEEING AND UNDERSTANDING WHAT IS SEEN 


The long range overall goal of basic studies in vision is the creation of 
human level ability In computers that dan understand describe p and search 
for things in images* To the extent that processing visual information. 
requires intelligence, nur studies of vision help us to understand, more 
sen oral issues. We have end will continue to learn a lot about problem 
solving in general by studying vision problens lh particular. 

It must be understood that the overall goal of creating human level 
visual competence has proved difficult, At first the critical path subgoal. 
ieetned to be the very prnblem solving issues we now know much about. Now wo 
feel, On the basis of considerable experience, that the key is the 
extraction Of as nuth information as possible using relatively knowledge- 
free techniques. The purpose of this section Is to defend our point of view 
and to break the critical path subgoa'j of image-to-symbollc-description 
translation into milestones and forecasted dates. 

Creating Computer Visior li JV o r u 1 

The study of conputer vision began very early in the history of artificial 
intelligence. Originally the problem of seeing and understanding what is' 
i&eti was to be a testbed for working owt deep problem solving and prograe 
organisation issues. Eventually this testbed purpose was served, but only 
after overcoming many frustrations and solving many unexpected problems - 

S Even blocks, world configurations produce complicated images, 
confusing programs with mutual illumination effects, 
indistinct interior edges, misleading scratches, the large 
variety of vertex configurations, end so on- 


I Imago Input devices like vidiccns and image dissectors dp 
not product the Simple clean images one might suppose -- ane 
must understand and account Ter variations in intensity 
sensitivity across the Image,, errors in image coordinate 
information, electronic noise, light source hum, and lack or 
dynamic range - 

Vision, it was discovered, is very hard, 

We tfewe Scratched The Edr fact 

To establish a content for what we propose, we lltt a few major steps 
forward in vision research. This is not intended to be a complete listing 
of achievements over the years — progress reports are the proper vehicle 
for that. Instead this is a sketch showing the shift in approach that has 
resulted from, cur progress. Vision is no longer naively thought to be a 
matter of a smart pro-blen solver looking at e few salient image points 
ins toad it is deaf that knowledge representation, constraint exploitation, 
and euro with low level vison are all critical, 

moreover, it has become clear that the low level visual processing we 
talk of will lead to techniques that demand a great deal of computational 
power. Thankfully IC t®chnology is progressing very quickly, so that a reel 
time, flexible visual processor is becoming a technological possibility. 

Given the research described in this section, it should be practical to 
look at photographs of buildings and describe them wall in five years. This 
research into basic issues that confront general-purpose visual processing 
will COmplemeht what we are doing now by way of special purpose vision, 
which has already begun to produce the prototype level results described 
elsewhere in this proposal. 


Lf? JrjQU Cofli t ra i ftt Expt Of iniica t$ The Right Uci/ To Deal With icrne 
Problem OJ fljraiflsij 

Waltz 1 work on understanding scenes surprised svaryone, Previously it was 
believed that only a program with a complicated central structure and lets 
df explicit reasoning power could hops to analyte scenes like that in 
figure S, fcciw we know that understanding the constraints the real world 
lopuses on how boundaries, concave and convex interiors, shadows* end cracks 
con come together at junctions is enough to wake things much simpler, A 
table which contains a list of the few thousand physically possible ways 
that line types can come together accompanied by a Simple matching pfagran 
ore all that is required. Scene analysis is translated into a problea 
resembling a Jigsaw puzzle or 4 Set df linear equations. No deep problem 
Solving effort is required^ It IS JUat a matter of executing a very simple 
Constraint dependant, iterative process that successively throws away 
incompatible line arrangement combinations, The implications of this type 
of analysis are far reaching, not only in vision, but also in natural 
language, it is now apparent that tho important task of determining the 
rolo of a noun group in a sentence -- agent, instrument, destination, 
conveyance, or whatEver -- can be done by just such a constraint process 
exploiting properties nf word order, prepositions. Individual verbs, and 
global context. The Idea la a key elenenc of Ptarcus 1 wait-and-see- sentence 
parser, laontioned in APPLICATIONS PffJWftf MS/C STfffflCS and described In an 
appendix. Ever, common sense electronic circuit reasoning has been shown by 
£ussman to use a Similar process. His work is also described in an 
appendix. 





















We JTflcw fftat finders tending The ffalAametlcj Of 1 Uumlhatfeu Efface* The 
Creation 0/ fmyjo.rtcr.rrE ^JioiifsiJ. Progrann 

To understand the real world t we must have a different set of primitives 
frwn the relatively simple ling trackers suitable and sufficient for the 
blocks world, In the past f imago intojiiitie-& have been used primarily to 
segment images using ad free methods based on differences In average iuage 
ill tensities or soma higher order measure. This is not ff&Otl enough. In 
order to deal with prgblen .5 la latge- analysis it is important to understand 
how Images are formed., It is not sufficient to try myriad processing 
techniques borrowed from other fields in the hope that one can be found that 
does something useful. 

Instead one should realise that images carry a great deal of 
exploitable information about the throe-dimensional nature of the surfaces 
of the objects imaged. Horn has demonstrated this when ha showed how to 
unwind the differential equations of image illumination in a way that allows 
shape to be recovered from shading gradients for simple reflectivity 
functions, later on. working with Harr, Horn goner allied a notion conceived 
by Land and showed that the natural reflectivities in a jumbled array of 
Colored papers can be determined despite complications from uneven 
Illumination which varies across the image. 5uch wart must ho continued. 
We must dev a lop an understanding of the nature of nnr visual invlronuent and 
the inage formation constraints if we are to successfully model what is 
being seen. 

We Jin ocr ^RDuriedpe Ja rtn important Adjunct To Image Proceai Cff Tec AfiliffsreS „ 

Figure JG illustrates the action of Shirai’s lino finder, representative of 
the work applying problem solving and control techniques tg Image level 
vision. Its fundamental thesis was that knowing a lot about the objects in 
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a scene me lies it possible to analyse the senna with primitives that 

otherwise would prove inadequate. 

Shlrai'l program knows, for example, that the objects it worts with are 
typically brick, shaped and produce many parallel lines. It also knows it 
must expect aligned objocts stacked on one another. Such facts enable it to 
guess at the location and direction of faint interior lines after it has 
traced out the strong background lines. Understanding the world should help 
process real images as well: knowing the approxiuatB place and nature of 
some Clear features like the doors and windows in a room should help locate 
walls, clocks h pictures, and so on, 

ShireL argued that his system illustrates tome rules of program 
organtietion and style known popularly as "the heterarchial approach*" 
Lately the development of heterarchial philosophy seems to have reached a 
natural plateau With the work of Freuder, His thesis, submitted in June, 
1975. develops heterarchial control structure ideas to new levels by 
presenting a collection of ideas about how systems can make effective use of 
Suggestions and advice: a hammer head, for example, proposes a possible 
location for its handle* Such systems have been much talked about and 
sometimes implemented in simple ways for sinple situations, but Freuder's 
result is a theory rather than an isolated illustration, Fretider develops 
such ideas to the point where it seems unlikely that further progress cm be 
cm do in the problem-solving/heterarchial control direction without vastly 
strengthening the processes that translate image' intensities Into the 
symbolic assertions required by any sort of reasoning prag rum. 

We hefteire more work bit COfllroJ structures /or Vfslort util not make 
sense until our irjtderstaridlrtp 0/ lou level Sirriolic processing celO-hes up, 
Tho real world differs from the blocks world significantly because there is 
a need for much better analysis between the low Level feature analyzers nnd 
the high, level problem solving, Via turn to this next. 


ue Kn$v> That The Translation From Intensity Arrays 7* Symbolic 7 0 pop rujUfttf 
Maps Is Far More Important JilOfT Was iJriginirirg fAOUffftt 

To stej out of the blocks world Into the real world requires more than 
problem solving, heterarchy, high levs! constraint, under*tending* and 
Is-olaied tmago processing techniques, Past progress In artificial 
intelligence it a necessary but not sufficient foundation, for supporting 
image understanding systems capable of analysing and describing pictures. 
We have cone to understand, that human quality vision requires talents not at 
first anticipated. It is these that we propose to work on now. 

Our central thesis is that if we are to understand how to make a 
machine see, we must get our hands the whole way around the problem, not 
just the so-called "high level* parts of it, tie know this in part because 
#fforts to rely on the high lava! work have not produced satisfactory 
results, whereas new work on low level vision by Karr and his students p dona 
with a view toward miking each level in the analysis as self-contained as 
possible, has sparkad excitement. These results and the proposals that grow 
out of then fom tha core of our planned work on basic studies in vision. 

A Solid Theory Oj Uistoji Ts, Comirrp 

Kara, in synopsis, is how Harr fsels about vision; 

■ The function of early visual processing is to compute a 
description of the gray-level changes present in an image in 
turns of a vocabulary of gray-level change primitives. 

These primitives consist of straight contour segments -of 
various kinds {SHAD INC-EDGE, EXTENDED-EDGE etc, P > t LIVES, 

BLOBs, and of various parameters bound to them such US 
FUZZINESS, CONTRAST or LIGHTNESS. POSIT SCW, ORIENTATION, 


simple measures of thoir SIZE, and & specification of their 
TtKhiMfiPf, points, This prlaitive descript lari Is obi a mod 
from the intensity array by knawledgE-frEe techniques, and 
it is called the primal sketch, The illustration in 
figure 11 hints at the density of the information taken from 
the image and represented In the primal sketch. The primal 
Sketch differs from an array of feature points in a subtle 
tray* which is explained fully in Harr's appended paper. 

i From our ability to interpret drawings, one may infer the 
presence in our perceptual equipment of symbolic processes 
that are capable of grouping linos, points and blobs 
tuuether in various Mays. Hon-symbolic techniques, like 
examining the power spectrum of the spatial Fourier 
transform of the drawings, cannot account for these grouping 
phenomena, since the groupings are performed by mechanisms 
of construction rather than median ism Of detection. 

■ For most mages, the primal sketch is large and unwieldy. 
However, it can be capably analyzed by t aechanism that has 
available the symbolic processes discovered in the previous 
stag, together with the ability to select item out of the 
primal sketch on the basis of simple first order 
discrimination* acting on the principal parameters. Hence, 
it Is argued, texture vision rests on grouping operations 
together with first order discriminations operating on the 
primal sketch, rather than on operations on pairs of points 
in the intensity array as Suggested by JUlesZ. ft Is 
further argued that the set of processes whose existence is 
necessary in order to explain our ability to interpret 
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drawings is also sufficient, whon applied to the primal 
sketch, to explain the range of texture vision that Is 
present in humans. Fourier and power-spectrum techniques on 
their own ere certainly deficient, and probably also 
unnecessary. 

Certain fish, incidentally, can mimic their surrounding environment 
vary wail with an adaptation time of only a few seconds. Harr's texture 
theory has led to a theory of the control of this texture camouflage. 
Presumably it depends on measuring and matching the small number of global 
texture statistics that Harr believes defines texture spaca, This theory Is 
being tested elsewhere at fl.I.T. under the direction of J. Y, lettvin. 

i, The extraction of a farm frea the primal sketch using these 
techniques annunts to figure-ground separation. Except in 
difficult cases, this extraction can proceed successfully 
without calling upon higher leval knowledge, and it precedes 
the description of the shap e of the extracted form. 

This has two important consequences. First, Lha isolation and delivery of a 
fora tu subsequent processes does not depend on being able to assign an 
accurate high-level description to It] and second.,-because of this it is 
easy to compute rough descriptions of complex forms. This is probably 
essential for the fluency of subsequent shape analysis. 





i The extent to which Higher Level knowledge and purpose 
influences the processing up to this Stage Is very limited. 

There is at present no reason to believe that higher level 
knowledge is needed to conputs the primal sketch at all; 

Its role in the extraction of form frcxi the primal sketch 
often can bn United to deciding which fora should ba 
extracted. 

It is conjectured that in ell cases,, higher level knowledge need be only 
weakly coupled to the processes that separate figure and ground. This 
relegates the use of higher Level knowledge to a much later stage then is 
found in current machine vision programs, and simultaneously confines its 
impact to influencing control, rather than interfering with the actual data- 
prccessing that is taking place Lower down, 

Karr 11 a appended progress report explains the theory in more detail, 

frojrejj fit I'tsioe firvotyei Severnt <3o-c?i j 

■ Complete the implementation of the ion- and Interned!ate^ 
levels of the visual theory outlined above. Rewrite 
grouping code for the sake of speed and assemble the whole, 

Large collection of routines into a single package. The 
grouping processes developed so far involve SCI. interesting 
mixture of local analysis, together with an ability to 
generate one or two more global Image variables. 


■ Eavisci and laplenent „ tho&ry of lightness cominitation that 
works after the extraction of contours. Various motivations 
flkist for this, partly difficulty In doing it at retinal 
level I partly the feasibility of doing it at this stage 
provided that (as now s«F15 possible) the image is divided 
Up into regions using knowledfle-free techniques, 

■ Increase the dimensions of the image that we cap handle- At 
present, our images are 1.-20 by 12B elements, which 
corresponds to Viewing an area 1 inch square at a distance 
of about five feat with the human retina. Even this small 
image stretches oiir resources, and processing one such image 
can take 20 or ifl minutes of CPU time. If we spend 
considerable erfnrt on re-writing chunks of the low-level 
code, we could probably move to a 250 by 256 image with only 
a small added cost in processing tine, 

I Put BLOBs securely into the primal sketch, h BLOB is a 
small, indivisible patch whose lightness differs from that 
of the surround. Although it is clear that BLOBs should be 
a primitive eleoent, the listing techniques for detecting 
these, from the output of the orleniaMon-dependsnt analysis 
are somewhat elementary, 

■ Devise a theory of the alignment Of an axis to an extracted 
fora. This is vital for the ideas we are currently 
developing about the transition from two to three- 
dimensional descriptions. One subgoal here is to devise and 
implement u theory qf symmetry detection because this is one 
way Of providing an axis. 


■ Carry out experiments to discover whether the theory 
provides a complete account of texture discrimination 
abilities, This is a difficult and arduous undertaking 
Since wo are really searching for a COUH ter - e nt-amp le . But 
analysis of (say) a hundred images□ chosen for their 
awkwardness* would go a long way towards establishing 
adequacy. At present, about 10 have been p races sad, and not 
all our texture-processins facilities have been used. 

I Assemble 4 C0rvpr Given live report nf the complete low-level 
visual processor. It would initially be published as a 
Technical Report, then a published monograph * 

b'e wiih to express fa tka stroflpest posiiiili terras our ufem that it is 
unroot 1st it to expect use/ui results jfrom most com port* rtts 0 / u phata- 
intmrprata if on project urrtti at ifdst tDIa iereJ 0 / ! ojser-^J eve! 
ficcheoloffics i! competence has bean dollfeved, cannot help hut feel that 
those who believe all parts of this iomenscly difficult problem can be 
solved using existing techniques and machines hove an insufficlont 
understanding or the magnitude of the issues that are involved. On tho 
pther hand, we do feel that a careful and energetic basic research approach 
to tba vision problem, in which one addresses the issues that arise as they 
bocope accessible for solution, is the correct long-term scientific 
approach. Problems will not he solved just because one wants to solve' them* 
they alto have to be ready rar solution. If we pursue the fundamental 
issues with diligence end vigor, then we nay reasonably expect that Within a 
period of about five years, some of these very advanced problems will Indeed 
coqh into development, rather than speculation range. But hear In mind that 
this cautious statement is in fact extremely radical, and would hevn to have 


regarded » wildly over-optimmic as little et ore year ago. 

#f3Jt 0/ Our Stated Objectives VIit Requite Toto fears 

Barring unforesBen delays, we should be able to complete much of this 
proposal within two years,. Many of these problems bow require labor' more 
than ideas, and unless we have overlooked something Important {which Is 
possible), w* should have completed the low- and intermediate-level studies 
by tho and of 1577, 

There are, incidentally, some problems we would like td solve but 
cannot approach yot. These problems Include those of understanding the 
visual dualities of gloss, glitter, sbinyness, wetness, metal, reflections, 
brilliance, transparency, and translucent, Ve feel that aany of these 
attributes lh in image can be detected using simple autonomous techniques, 
but as yet our grounds for this ara speculative, reitihg mainly on the 
intuition that they generate recognizably distinct 'qualifiers* in a 
perception, rather like color does. UL1 can’s study of fluorescence 
strengthens this feeling because he showed that an autonomous, knowledgo- 
free nethod can detect light sources, 

fftfaiinj AbQttt A Vision JTacaine fs 4 Geaeriri ffedien-fe m (To Ql 

Once a system has been in.pl Emeu tad in a single, autonomous package, we shall 
bo able to study highar*level (3-P) issues in the context of what such a 
system can deliver from raw image data, The sheer computing power that is 
needed to carry out testurs processing on an image of reasonable size means 
that we should begin thinking about designing same special^purpose hardware 
that is powerful enough to carry out the processing in a few seconds. 
Accordingly, wo have begun preliminary design studies on such a machinej and 
we wish to continue these. The two prerequisites for serious .proposals for 


construction are (1) a firm, demonstration that the PDP-JO- implementation of 
the whole proce-Ssnr performs satisfactorily; and (11) that OUT design 
studies result in a solution tn the hardware problems that lias within what 
i% technologically possible, and which will perform tin? whole operation 
within 20^0 seconds; and (lii> that the expense is reasonable and nothing 
like the multimiLlion-do 1lar parallel machines of the past. Unless 
unforeseen difficulties arise, we expect to have a concrete design proposal, 
together with proof of its processing adequacy and computational prowess, by 
aarly 1977, 

COMMON SENSE REASONING 

It Is not possible to feel comfortable with the problem of common sense 
reasoning, Progress has been made by Sussman, Goldstein, and others like 
McCarthy at Stanford and Newell at CameI?i&-Me 1 Ion, but sene very basic work 
remains to be done. 

Since -Slagle's integration program helped establish artificial 
intelligence as a serious science, researchers have mined nut mere and more 
narrow domains by ferreting out what kind of knowledge is involved and how 
to use it, A correct complaint rests an the observation that one program 
contributes to the next Only -on the level of ideas and methodology. 
Something better is needed. Collecting a group of specialist programs 
together for a particular kind of problem is good and helpful but not a full 
solution. 

Thinking about common sense reasoning see-mi to be the right thing to 
da. This is not * new- theme, nor is lack of a better theory an obstacle to 
the First round of progress on the Very Large Data Base preblets or the 
Intelligent Terminal Problem, bn the other hand, we believe that without 
basic research in this area, progress on practical problems would stop short 
Qf a full solution. Systems would tend tq be Constrained to the specific 


chosen implementation domains with transfer to other areas redlining 
difficult and time consuming. A systen warned out for medicine wouid bo 
useful to the creation of a system fgr ship maintenance, but much now work 
on the epistemology of ship nalntenanco would have to ha force fed. 

But what Is the alternative? We Oa(iflre tftfft waders tending COrtmOfl 
jtAir reasoning fs necessary before systems Co n £e c reeled uAich loom 
oasiifl eftotit specfaiiied problem splninp domains. Trying to work out 
mechanisms that draw direct analogies between heart disease and Che 
maintenance of boilers mioht he one approach- But to us, a hatter idea la 
to discover how both might be understood in tores of the epistemology of the 
simple, everyday physical world. We therefore bellene o lay need is a 
tftec ry 0 / cowmen .sense reasoning cixurl simpie pAyj{cat meclienisms- 

^USSnvetV and Goldstein have made a good Start by working oat notions 
dealing with process: we now have a better vocabulary for talking about 
goals, plans, end the structure of programs. What we need now is a 
complementary effort focused on (Mir physical environment. We need to have a 
better vocabulary for talking about objects and materials^ We need to apply 
our classical methodology to the simple physical world interaction problem- 
We need to establish what kind of knowledge is needed, what specifically is 
needed, and how much. We need to work out the epistemology of interacting 
things -- things pushing ntia another, the idea of filling, filling to store 
*bd filling to inflate and filling to the bursting point as in hearts and 
boilers- Our systems tiaed to know about the ideas or repair, patch, and 
breakdown, the toughness of common materials, the Idea of floating, rolling, 
mating, helping. 

The observations of Hartin at HIT'S Project HAC and of Schank at Tala 
reinforce the idea that specialized reasoning is enabled or at least 
facilitated by analogy to simple physical happenings. Everyday language is 
replete with physical world terms and usage applied to abstract facts end 
processes. In the mental world we talk of ferreting out facts! in Partin's 


social world ownership moves from place to place with language spited 
equally to objects; In the logical world facts support one another in a Hay 
not unlike Vinograd's blocks; and for a typical expert problem solving' 
domain. It is rare to think of learning electronics as a child without 
thinking about resistors In terms of narrow water pipes and capacitors In 
terns of water balloons or some such reservoirs. 

Given such a world theory we believe we will be In much better shape to 
handle all sorts of diverse fields„ At first the system may not conclude 
for Itself that a heart can be tinder stood in the same terns used to 
understand the atomic vessel and fluid mechanism, but if we can tell the 
systcr. such things, much needless reinstruction and narrowly focused codihg 
would be circumvented. 

There Art A tfumoer Qf Geetf Precede rata For fleirtflopiirg A TAeorg Of Comraois 
Sense Reasoning 


H The Ph-D. theses of Sussman and Goldstein probed the 
epistemology of processes, showing that considerable 
regularity emerges when the right descriptive apparatus is 
available. Sussman worked with the automatic debugging of 
blocks world motion primitives a ia Winograd f and Goldstein, 
handled simple geometric figure programs. 

■ Susssar*, Brown, and Hcltermott' s work on electronics cerri.es 
this thread forward and gives us considerable understanding 
of procedural aspects of the problem and of the interfaces 
between complex parts- 


i Schemas conceptual dependency theory, although * bit 
ovcrfocused on the human body,, is als-s an intellectual 
precursor. Thu important feature is the demonstration that 
con vine tag broad-based forma Li sms are achievable, Ve expect 
our theory to bo considerably more complicated, but not 
impossibly so. 

i papart uses the term "Qualitative physics' 1 to cover the 
problem of understanding the pendulum and the gyroscope in 
simpler., push-and~puil terms. In his IV’SF supported work, in 
education, ha has worked out ways of Looking at these things 
that will be helpful - 

We Plan fo Attack rfte Comrufl SertJC HeaSGfltfljJ fret [ear From Four directions 

in summary, the long range objective of our work is an understanding of the 
simple physical lateractions of staple physical things sufficiently good to 
serve os ad analogical foundation for programs that wanS to reason about 
more abstract problems- (For full power, the analogy process eventually may 
Halit to Do capable of working between two specialiied domains as well ftS 
hack to the fundamental physical One.) 

W« believe this is deep research and some breakthrough requirements 
prevent us froo giving honest estimates of advanced perfornance lev*Is and 
dates . We do bo Hava, howevor, that there are good ideas about the next 
steps and that wort on these in tha two year time frame will leave: us ready 
to make the predictions that would be specious now. 

In particular, wo faOl that the nest steps must be directed to work cun. 


those four subjects: 


I Frolllng wants to understand the priaitivss that are 
prerequisite to understanding basic machinery. He talks in 
terns of shafts, surfaces, wedges, am) a dozen or so other 
similar primitives. Kb further aenlions interactions like 
rolling, sliding, and the like. And putting everything 
together he wants a system that can understand how a camera 
works by drawing analogy between light and fluid flow,, 
realizing in consequence the need for a container for the 
filn, a condo it for the light, and a part to control its 
"flow-* This is aobitious, and two years' work may not 
produce such an advanced scenario, but the direction seems 
correct and some analogical reasoning ability seems certain. 
^rolling's notes on approach are appended. 

Z DoKlecr's study of qualitative reasoning complaments 
Frellliig's by attacking the question of how to deal with the 
boundaries that segment complicated problems Into Slaplft 
pieces within which straightforward mathematical treatment 
suffices. Reasoning of the sort Be it leer is getting «t 
pervade our thinking whenever we nust have more than a rough 
idea about bow a complicated machine works -- surely it is a 
prerequisite to design and must bo worked out if machines 
are to be designed automatically. 


3 Richard Brawn urates the general inflation from A third 
direction, that of concentrating exclusively on using 
analogy in reasoning. His i ppro n.c n is just non taking shape 
and now it can be said only that solid geometry may be the 
domain because a great deal is known about solving tough 
solid geanetry problems by using Simplified plain geometry 
problem; together with analogy. This domain will be pursued 
only if it remains promising as a domain in which to study 
the genera} analogy problems. The ijeomeiry, per SO, is of 
secondary interest- 

A Eulpors it working on representing the kind of knowledge 
that peoples have in their mental maps of cities. This is 
the knowledge which is used to represent where things are 
with respect to each other* how te get from one place to 
another, and what kinds of groupings they fall into- His 
research will shed considerable light on the issues of 
imagery, visual memory, visual problem solving, and sini Jar 
Issues Involving the internal representation of Spatial 
knowledge- There Is considerable psychological evidence 
that for humans the mechanisns used, for representing and 
manipulating such knowledge is extensively reused when 
dealing with aoro abstract topics like the world of 
oraanimations, Kulper"; project complements nut only the 
program aimed at dealing with common sense reasoning but 
also the work of Levin which is oriented toward extracting 
information from scenes with hills and lakes. Levin’s 
proposals are described in WFlTCflnoxs ORSEVTlB U1S10H AMD 


MAHIPULATI OH- 


EXPERT PROBLEM SOLVING, LEARNING, AND THE 
EPISTEMOLOGY OP ENGINEERING 


A lung range goal of Artificial Intelligence research it to make a system 
which Cftfl assimilate new knowledge and Leara from experience* In reaching 
Tor it we have seen many attempts at producing universal formalise**- Each 
of those huS made the fundamental assumption that knowledge is additive; 
that means Cftri be round far assimilating new knowledge Without overhaul of 
the existing knowledge structure. People who design problem solvers are not 
alone in making this fatal assumption; many students believe that they can 
Inarn, by just listening to facts thrown at them in lecture hells and that 
learning can happen without much intellectual effort. The truth is that 
learning involves great effort. Introduction of a new fact or concept into 
a knowledge structure often precipitates upheavals and dislocations in that 
structure. Any system which can assimilate new facts must be competent fcc 
modify and manipulate Its existing knowledge. 

Engineers do this constantly,, they combine, analyze, debug* ahd, explain 
stuctures in the course of design. They decide hew simpler structures can 
be combined to achieve particular goals end they predict the behavior of 
composite structures by combining the mure primitive behavior of tho 
substructures out of which they are formed. Then they use analysis because 
proposal of plausible designs may lead tu design* which do not quite work. 
This is discovered by analys is and lead* into the debugging phase of design - 
Finally, an engine or HU St be able to explain, the devices which he has 
designed, An explanation is often a description of how the behavior of the 
composite device can be attributed ta the Combined behaviors of Its parts* 


Ite Vffirt To ffnrfor j Ut nd Prvgramies T* fernis QJ Entiineerifiji Pesijn 

Since same knowledge ll inherently procedural* and thus is best represented 
as programs * we are forced to the conclusion that a learning program must be 
a programmer -- it must understand kneuledgo re presented is procedures, sc 
as to bo able to modify and. patch them to incorporate desired changes. 

■ Consequently we must have a sound theory of programs and how 
their structure relates to their functinns as defined by the 
goals of their designer. He are ltd to a study of the 
epistemology of the design of programs. 

Can we write pro grans that combine, analyse-, debug, and explain other 
pro grams 7 We have begun to, but we do not know how to properly describe a 
prograra and its relationship to its design* Programs are hard to document 
In clear and useful ways, lo order to write programs that reason about 
programs we iust somehow learn to describe programs in terms of tha 
designer's goals and plans and the interactions of the parts. The research 
we propose to do Is aimed flt clarifying these concepts. 

We believe that some general program-iiing knowledge is composed of "plan 
fragments’ which are abstract and rather general plans tor accomplishing 
goals. An example from Sussnan's blocks world program writing system is: 
"To achieve x and y, try to achieve them separately." Another simple 
example is the fragment: "7o achieve x+ if we have a B&thod of achieving X 
given that we have y, first achieve y t then US* the method for x." Still 
Another Is: "To achieve l There are no x's With property P,' find each X 
with property P and achieve the negation of P on x." These rules are in fact 
only heuristic and likely to result in begs. Though the fragments shown are 
abstract, fragments are intended to represent concrete knowledge as well. 
Programs are constructed by expanding and instantiating plan fragments. At 


sacti level, which of the applicable plan fragments is used to achieve a 
particular goal is a design decision, A plan for a program Is, at least in 
pert, the tree of expansions of plan fragments, with particular design 
decisions, used to create the program. Since plan fragments often expand 
into bugs, we also need notations showing what constraints were added (such 
as ordering} which were not parts of the original design, and what hugs they 
were supposed to eliminate. 

ProprarlJ Are 0*e fcampLi Of The HotUn Of Deliberate Artifact 

It is probably clear that plans for programs can become very complicated and 
studying the epistemology of programs directly may be too hardi, Programs 
have conditional control structures -■ their configurations {.how they break 
up into parts and how these parts are related} nay depend strongly on their 
inputs. They have shared structures such as subroutinos and a modification 
to such a substructure potentially affects all of the places that 
substructure is used. Programs also deal with rather complex encodings aF 
information -- there are many kinds of data structure a programmer must know 
how to maoipulate, 

Thesa facts are discouraging, hut it has become clear to us that tho 
notions of plan frayaents and plans, and using such concepts in design, 
analysis, debugging, and pxplanation, are definitely not unique to the 
do-main Of programming but are common to reasoning about any "deliberate 
artifact ,* Ua have come to feel that we can make substantial progress in 
the Study of programs and their properties by first examining the nature of 
those concepts in other, simpler domains. 


T^ere Js ^ CoJitiniMM Of For Studyiiiltf J"fic fpitlsmoiojrif Of Deliberate 

Artifacts 

We have searched for 9 Simple domain ■Hhich eliminates many of the problems 
posed, by the Study Of programming but Which also deals -with complex 
del ihorfit,.-; artifact! c-ompesed of stapler artifacts deliberately arranged. 
In our search we wanted to avoid several difficulties: the- problems of 
spatial reasoning, visual recognition, and natural linguistics, all of which 
Involve other deep, deflecting issues. 

As we see It, nil Of this argues far diving into the following at a 
carefully selected point: 

■ The simple and ouch studied blocks world, 

■ One of the many engineering areas that can ba approached 
through the relatively simple mathematics of luaped 
parameter linear systems. Thesa Include the design of 
electronic circuits, many simple mechanical devices, and 
hydro He systems, 

■ One of the nany engineering areas that makes uS'ft of 
interconnections of subsystems character!t#d as ported, 
linear, black box information or materials processors. The 
lumped parameter linear Systems mentioned above are to be 
understood to be a subset of this class far which the port 
descriptions are constrained to a certain simple forn* 

Almost all forms of engineering havE a strong involvement 
with tbit Kind Of analysis In the course of design. 



■ One oT the areal ifl Which the black boxes are non-linear. 


I Program, 

In the work done so rar, the con cent ret ions has been on the design of 
lumped parameter linear systems for several reasons. It t* well understood 
by engineers of all sorts and it has a precise technical vocabulary along 
with interesting problems. The criteria for success are clear, the lumped, 
parameter elements interact In stereotyped ways only at designated ports. 
Evah flLBethaniCttl engineers r who are often stuck with -geometry* Still make 
use of these principals although they are limited in the comp 1 est ity they can 
successfully handle relative to, say, the electronics circuit designer. 

If rich Is hooded 

our work to data, executed in the study world of electronics design, 
has confirmed a need for the following obvious things: 

1 The need fur knowledge representations adequate to define 
system requirements. 

■ The need for transformations among representations, in this 
case magnitude graphs to pelt-tore pints to algebraic system 
function form, for example. 

a The nettd to match plan fragments in the bag of tricks 
against system requirements. There Is an implied need to 
understand hew to match when the matches are not exact- 


■ The need ta debug the undesirable interactions that fallow 
from first approximations which ignore fragment 
interactions. In this case, the loading of one stage on 
another, 

All these ere necessary ae heiieirc, to erpert problem joltftfltf in penerc! 
stare tflcjf ftose s ifsl 2 OSUes t J1 e/i^ rfomjfjf Eftal CPU Id he class i/^ed SJ e denrata 
□/ Oclihcratc « rt li/oCts . 

Figure M shows the kind of control structure we feel is required for 
the hinds or problems we have in mind. Hate that there is often a "Bag of 
tricks 11 -- a data base referred to by all parts of the program- 

Thvrt Is h Hen For Anders tend tag friibtrdte ArEt/ccts 

The Engineering Problem Solving Project is concerned in the short run with 
electronlc design (either IC or discrate). Later it is to blend into an 
attack on the structure of more complicated artifacts , such as programs, in 
this section, we outline our timetable. 











































I Gerald Sussman, Richard Stallman* and Gerald Eoylance have 
written and extended EL, a program for circuit au-alysis. 
This program uses Its fcnowlMge of the plan structure of the 
analysed device to avoid blind relaxation techniques in 
analyzing circuits. As of new, El handles quiescent circuit 
analysis. Bus small et i i arc currently engaged in extending 
EL to AC signal analysis and reorganizing It toward greater 
flexibility* By June 1976 they plan to have a finished 
product. EL's ability to "explain" its conclusions will be 
critical. The system Will not only give answers on demand, 
it will explain those answers as well. 


1 Currently, good progress is being made on electronics design 
debugging. Allen Brown has finished a Rti.D, dissertation 
on *Localization of Failures in JUdio Circuits." This wort 
Will be extended by Matt Mason, who will implement brown's 
ideas. He should have Brown's LOCAL program coded by June, 


1 * 76 , 


a Drew HcDarmott is studying the problem of Interface design. 

He has written 0 wary slnple program, that knows about a few 
Of the linear system plan fregaenti. Thera are two DOin 
subprobleas that have arisen: choosing transforaation 
techniques for problems; and satisfying conflicting 
constraints in putting plan fragments together. He will 
complete a PJt.D, dissertation on these topics by June 1976, 

HcDeraott Is doing his research under a commitment to a 
certain Lind of modularity in programr,ing. He would like 
his prosraa to have "wired in" the theory of artifact 
design* but the plan fragments* problao. analysis 
techniques,, and other "advice'' are to b# more loosely 
coupled to the program. 

I There regains the goal of understanding the design and 
modification of complex programs. In parallel with our 
electronics work. Charles Rich and Howard Shrobe have been 
studying computer programs directly, in terms of eh^Uieerod 
artifacts * Ilka circuits. They are aiming at a progran to 
aid a human in designing hash tables, by June 3S-7&. 

■ The Rich and Shrobe work is pert of a Larger goal of the 
development of design analysis tools for programs analogous 
to the analysis tools used in lumped parameter and linear 
systems analysis. 

For reasons we have mentionedi programming is a very difficult domain - 
It is hard now to predict what we will have learned from another engineering 
discipline, in a year, we will be in a good position to bring our 
intuitions and experiences to bear on program design- 


Wc bel ititt tfrdt the ijjrt ouliinei i,n tbit iesilO/! U pap c onpo nr o ! of 
4s jrecej-surjf before Cdfl desfpn systems tJlni can roe!Itf writ* their 
pun programs. Ihe other Cftnponant needed, it seems to US* is the thorough 
understanding of the domains for which the automatic systems ere to 
programmed, this is one reason Why we applaud dartin's wort in Project HAC. 
We believe opr effort to ho cottipletnoptary tP his. 

PERSONNEL 

■ Work In baste vision studies is under the general direction 
of DAVID HARR, His theories of vision are on the critical 
path to a batter understanding of vision. Professor 
BERTHOLD K. P. KG PA p whos® first responsibility is to direct 
the applications of vision and manipulation work, will 
continue tn contribute from time tD time. 

1 SHIMON ULLHAhl, who just recently coapleud a sucessful 
study of light source detection,, will continue to pursue 
aspects of the Barr theory. Other students with likely 
involvement include KEN FOAEUS, an undergraduate who has 
helped Barr with programming for some time and JAN 
GALK0W&£It a graduate student who has just completed a Study 
of electronic component identification. 

■ Professor PATRICK H, WINSTON plans to taka an active rols iti 
working with students on the cpiman sehse physical reasoning 
problem. 


■ M1KE FREILING has proposed to work an the eosinon sense 
reasoning and analogy ntdianlsns appropriate to basic 
toachinary, RICHARD BROWN, hoping to deal directly with the 
problem OT analogical reasoning,, will try to expose key 
issues by studying the way solid geometry problems are 
solved throgh plane gaonatry analogies. JOHAN DEK.LEER, 
working under the supervision of Professor SEYMOUR PAPERT, 
is working on the fluaiitativc knowledge/quantitative 
knowledge interface. BEN WIPERS is doing mental maps, 

■ Professor GERALD SDSSJ1A.N is in charge of making progress lrt 
the general domain of expert problem solving., prograta 
understanding systems, and the apistenology of "deliberate 
artifacts," Soma of this work is dona in cooperation with 
Professor CARL HEWITT, supported mostly through Project MAC 
and partly through the Artificial Intelligence Laboratory, 

■ Susspan and .Hewitt are Jointly supervising CHARLES RICH and 
HOWIe scHROBEj wfoo are working on the problem of expert 
problea solving using an expert In programning as the 
discovery domain, 

S Sussman has worked closely with RICHARD 5"ALLflAN and GERALD 
ROYLAfJCE on the development of the electronics expert, EL, 
Roylanee plans considerable further werk in this direction, 
expecting to introduce sJiftll signal analysis to complement 
the quiescent analysis expert implemented so far* 
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FREILING: OH SIMPLE MACHINES AND ANALOGIES AMONG THEM 

DEKLEEB: QUALITATIVE AND QUANTITATIVE KNOWLEDGE IN CLASSICAL MECHANICS 

SU5SJ1AN AND STALLMAN: HEURISTIC TECHNIQUES IN CIRCUIT ANALYSIS 
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SYSTEMS AND SYSTEMS CONCEPTS 


The Artificial Intelligence Laboratory wishes to develop a lew cost system 
intended to serve single users with an outstanding LIS? together with 
equally high quality text editing, debugging, and display features. The 
overall goal is to produce a computing tystea dramatically superior to any 
available today both in terms of power and cast, Ue intend to achieve this 
goal by way of implementing a prototype system manifesting ideas now worked 
□ ut and detailed ill appendices, 

We also intend to do some thinking about vision processors with a viaw 
toward preparing for the time when sucti machines are both necessary and 
possible. 

PURPOSE 

This section is short since other appended documents fully convey the 
general spirit and technical detail behind this effort. These few words 
will simply summarise the salient points - 

Tide LISP flffcJitnc Promises A A'em Way Of Thinking dioct ConrpuUflp 

The trend has been toward ..cheap hardware and expensive software. In 

contrast to the classical shared CfrU and central memory systems, the LISP 

□ach.ine COnC apt eX ^lolti the Cheep hardware treed. A principal_re s uit of 
the overall approach Is simpler system software. The impact on how 

co-nputlog is done will he, we believe, at least as dramatic as the 
introduction of tine ^sharing. It is perhaps not surprising that such a 
development should spring up from the conraonity of people who think about 
Artificial IotelllgeoCe inasmuch aa Artificial Intelligence development has 
always made extreme, technology j pult demauds on computer system science. It 










is- argued, after *11 1 that time-sharing it self was largely incubated and 
promoted by early work in artificial In tell i genca. 

me tfsp Mini Offers More Power 

The family of ideal that make the USP machine an exciting prospect 
anticipate the future, PartiraQiint among these ideas may be the large address 
space offered. We believe that of address Space will be naeded 
eventually in situations where natural English input is demanded. We base 
this on. estimates that seen sensible in terns of Winograd's natural language 
work in the simple blocks world which required 15DX and on subsequent work 
by Martin which leads him to talk in terns of 51Z£ ns a minimum. Existing 
and proposed PDP-111 type machines allow only lfl of the JO bits of address 
needed, while the LISP machine is to provide Z3, a comfortable margin, (The 
JCL-1Q processor will address more then IS bits of address space, but only by 
way of an awkward aftarthought method forced by design decisions which were- 
reasonable at the time the PDP-1D architecture was introduced years ago.) 

We believe that natural language is not alone in suggesting the need. 
5 d far programs have worked with sophisticated programs on small data bases, 
As wb became mare ambitious and more concerned with real problems, the 
larger address spaces will be not just desired, but essential. We note, a$ 
an example of this, that the CCNNIVER interpreter alone occupied over 100K. 
Of central mEmory and thareforo points toward systems containing it which 
Bust be beynnd the ranch af the JS'frKj IB hit address space, 

LISP Machine ffeVCi opmeRt WiU £e Sirrprf st ir(r l Iltexptntlve 

Ten years age it would have been foolhardy for a university group tq propose 
the creation of a grand new computing engine. Times have changed. Large 
seal* integration has given us prefabricated walls where before them was 


only brick and mortar. Five years ago at least ten tines as many chips 
would hav® hoot needed. Mo re over, 5 van. Ford and IU.T. work on interactive, 
graphically oriented design aids has given us autor.atical.ly generated 
documentation and wire wrapping programs where before we bad only drafting 
mistakes i lost hits of paper, and chaos , Figure ]J Illustrates the sort of 
documentary output automatically generated in the cuurse of the design' 
process. These new tools have been thoroughly demonstrated through use on 
projects Of demanding scape, including the design of the new DEC K.L-1 D 
processor. 

The LISP Machine Will Serve The Heeds, QJ Hatty 

The time pas coma to produce such a machine because the technology is ready 
and the users breathless. Certainly the artificial intelligence USD 
community is the first source of custamers, Far this group, not only are 
the technical advantages ext rent, hot also the new ability to get started 
with an initial Investment one-tenth the size of a PEF-IO configuration Is 
extremely attractive. Smooth expansion via one-bygone addition Of LISP 
machines substitutes for big ticket decisions. Hare people, widely 
supported, will do Artificial Intelligence work. Of course the scope of the 
predicted Impact reaches far beyond Artificial Intellicence for two reasons? 
one is that the predicted Infiltration of Artificial Intelligence into most 
or the frontier problem areas of computer science has occurredi, another is 
the fact that the LISP machine itself, while worked out with LI&P in mind, 
will certainly support other languages with largely the same advantages.. 
The concept will make life easier and more cost effective not only for the 
Artificial Intelligence LISP using conravnity, but also for those who for 
historical or genuine technical reasons nust stick with the more classical 
languages for production and execution of end user software. 
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time, wtu be tAe JWrdiPirre for tile c^wrjuter user'a INTELLIGENT 

TERMINAL, lJl« miiflflpor'j MANAGEMENT ASSISTANT, the atudent or trainee’* 
PERSONAL PEPAGOGUE. the factory's INTELLIGENT ASSEMBLER, and, ta general, 
ei/mri/one's PERSONAL HELPER, 

THE LISP MACHINE CONCEPT 

The hardware confIguration is diagraiiffled in figure 14, It is to consist of 
the following corn: 

1 A FLIP-11 for doing I/O, for controlling a unifeus, and for 
Aiding in debugging Other components, principally the CQN£ 
processor r 

K The COWS processor for executing LIS? tailored Instructions. 

The CON'S- processor successfully integrates many good ideas* 
among them programmable microcode, LISP oriented byte 
handling operations, special list processing directed 
addressing modes, a fast internil stack, and more. The CONS 
machine is fully described In an appendix, 

ft A good deal of central memory, probably on the order of 50K 
to 1QM H bit words, This large amount of non-aharod, 
dedicated local memory is determined by the overall design 
philosophy which rejects classical time sharing in favor of 
system simplicity and performance- 
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■ A fast Ideal disk or possibly CCD bulk, senary of the sort 
recently heralded by the Introduction of let CCD chips. 

This will hold that part of the Z3 bit address space not in 
•fast" meaory and will be the primary source of system 
software and user programs, 

■ A connect ion to the rest of the computing world seen by the 
machine. Here is where the sharing horizon Is to stand, 

Some link, probably some sort of Ether net, connects users 
together, provides for data and program sharing, and all the 
usual cooBunlty features introduced by time sharing, 

COMPUTING ALTERNATIVES 

Several other competing recommendations for meeting computation needs have 
been considered. There are good arguments for some although we believe the 
weight of the evidence favors the LIS? machine, Here are the principal 
Alternatives. 


■ Option Ji Sinply wait for the 15<M00 FDk-lO. The extensive 
software work already done makes this a reasonable position. 
Unfortunately the once state-of-the-art PftP-10 architecture 
now creaks with toe. The snail address space is 
particularly worrisome. Us do see, however, an intermediate 
time during which LI-5-P machines will stand parasitic on PDP- 
10 systems waiting for users ready to run LISP. The PEP-10 
will continue to run the excellent text editing end general 
support software already existing, while LISP execution 
takes advantage of the LIST' machine's dramatically better 
ideas for running .LISP- This is the reverse of the usual 
idea which forms out the secretarial work to PDF-lls, 
presumably running yet unwritten secretarial software, 

■ Option Z: Develop a PDP-11 LISP. We have done this, but 
intend no major work dependant on It, The address space 
available to the PDP-11 architecture forbids It, Snob a 
LISP can run some small demonstration and application 
programs, but only a small portion of those being written 
now by workers In Artificial Intelligence. 

i Option 3: Finish the LISP machine. This seems the wise 
Course. 

LISP MACHINE OBJECTIVES 


Our Intention is that tht LISP machine development be completed during the 
next two years with a prototype running by January, 19?6 and in advanced 
prototype running by January x 1977, 


LISP MACHINE HISTORY 


Richard Greenblatt eonc 0 ivpd the idea of the LIS? machine. He was soon 
working In close cooperation with Tom Knight. 

the work proposed Irnro was earlier proposed as a Joint project between 
the Artificial Intelligence Laboratory and Project MAC, Joel Moses of 
Project MAC wrote the original proposal, As now organized, however* the 
work will be handled within the Artificial Intelligence Laboratory with a 
view toward close supporting counsel and communication with Project MAC, 
which is to he regarded as equally r&spopsiblo for getting the ideas off the 
ground. It seems clear that HAC5YHA and other Project MAC interests will be 
best served in the future by the LISP Machine or something very similar, 

PERSONNEL 

This project is in the hands of people well known for their ideas tnd 
preeminence in system software and hardware design, RICHARD GREEKBLATT, 
principal author of the ITS time-sharing system, is in charge of the 
software, THOMAS K&llGHT, principal author of the knight terminal display 
system, innumerable interfaces and special purpose processors* and overall 
coordinator of our digital hardware- systems, is. in. charge of the hardware. 
They will work with JACK HOLLOHAf, wail known for his work on the so-called 
Superfocmly Project at Stanford which was extremely influential in the 
development of the latest ih the P&P-10 family of conputers, the K.L-S0, and 
with JQHhi PKHJ5S0URIS, who is now finishing up a parallel processor for chess 
{not funded by ARPA). BILL FREEMAN will interface the LISP Machine to the 
existing time sharing system, 


A VISION MACHINE 


Soon we will know enough and prices of processors will be sufficiently low 
for sensible proposals to be cade for very general parallel image processing 
configurations. We ere therefore giving* tome thought to what could be done, 
particularly With the recently announced LSI-Us. This thinking Is not far 
along and it Is inappropriate to make a sharp proposal at this time. We do 
append a working paper on the subject* however* to show that such a project 
deserves same thought. 

APPENDICES 

HORN AND WINSTON: PERSONAL COHPUTERS (REPRINTED FR0J1 DATAMATION MAGAZINE) 

HOSES: ORIGINAL LISP KACK1NE PROPOSAL 

KNIGHT: THE CONS MICRO PROCESSOR 


HARR AND FOR&US: VISION MACHINE — PRELIMINARY ASSESSMENT 


